Challenges 2019


"2019 BenchCouncil International Artificial Intelligence System Challenges" consists of system challenges (on many kinds of chips such as Cambricon, X86, RISC-V), and Intellifusion algorithm challenge.

The result submission extented to Oct 7.
Submission site: http://125.39.136.212:8090/


More details about AI Challenges are available from the challenge manual: http://www.benchcouncil.org/competition/handbook-en.pdf.

Awards
500,000 CNY in total, about 70K US Dollar.

  • Special Award (Only one): 100,000 CNY
  • The First Prize for Every Track: 30,000 CNY
  • The Second Prize for Every Track: 20,000 CNY
  • The Third Prize for Every Track: 10,000 CNY

Four Challenges Tracks

The topics of four tracks are derived from AIBench (TR, Bench18). Also, the provided models are trained using AIBench benchmarks.
Download AIBench (Hosted on BenchCouncil code management system---BenchHub):
Micro Benchmarks: http://125.39.136.212:8090/AIBench/DC_AIBench_Micro
Component Benchmarks: http://125.39.136.212:8090/AIBench/DC_AIBench_Component
Application Benchmarks: http://125.39.136.212:8090/AIBench/AIBench_DCMIX

  • Track 1:International AI System Challenge based on RISC-V
  • Subject:
    The implementation and optimization of convolutional neural network based image classification task on RISC-V.
    Requirements:
    (1) Implement the forward Calculation of neural network. Load and run the given trained model ResNet-20, the format of model is hdf5.
    Download simulator:https://hub.docker.com/r/crva/riscv-qemu
    Download data:https://www.cs.toronto.edu/~kriz/cifar-10-binary.tar.gz
    Download model: http://www.benchcouncil.org/competition/cifar_resnet.zip
    (2) Minimize external dependences, and try not to use external libraries (e.g., OpenMP, Pthread, Boost).
    (3) Use the model required by the organizer and guarantee the original model accuracy (84.03%). The accuracy deviation with the result of organizer provided model is within 0.05%.
    Metrics:
    (1) Minimize the binary file (e.g., the executable files compiled by RISC-V compiler, model weights, etc).
    (2) Maximize the execution performance and minimize the number of instructions.

  • Track 2:International AI System Challenge based on Cambrian Chip
  • Subject:
    The implementation and optimization of convolutional neural network based image classification task on Cambrian.
    Requirements:
    (1) Implement the forward Calculation of neural network. Load and run the given trained model ResNet-50.
    Download data (image):http://125.39.136.212:8484/Cambricon/cifar10_test.tar
    Download data (binary):https://www.cs.toronto.edu/~kriz/cifar-10-binary.tar.gz
    Download model: http://125.39.136.212:8484/Cambricon/model.tar
    (2) Use the model required by the organizer and guarantee the original model accuracy (84.39%). The accuracy deviation with the result of organizer provided model is within 0.05%.
    (3) The result submission should include source code, description document, and model accuracy.
    Metrics:
    (1) Maximize the execution performance and minimize the prediction time (wall clock time) on provided test data.

  • Track 3:International AI System Challenge based on X86 Platform
  • Subject:
    The implementation and optimization of matrix decomposition based collaborative filtering task on X86 platform.
    Requirements:
    (1) Implement ALS-WR training algorithm on X86 platform. The competitors can use external libraries supported by the platform.
    Download data:https://grouplens.org/datasets/movielens/
    (2) The parameter nf chooses 100. Training 30 rounds using Movielens dataset, shorter training time is better.
    Metrics:
    (1) Maximize the execution performance and minimize the training time (wall clock time) on provided data.

  • Track 4:International 3D Face Recognition Algorithm Challenge
  • Subject:
    3DFRC (3D Face Recognition Challenge) aims at soliciting new approaches to advance the state-of-the-art in face recognition.
    Requirements:
    (1) The competitors are optional to use external data for model training.
    Download data:http://125.39.136.212:8484/3dvggface2_1.tar.gz
    (2) The external data used for training must be descripted in "method description" file.
    (3) The competitors need to submit the model file and test file.
    (4) Source code for test and description document should be submitted. The source code for test need to implement related interfaces using python3 and c++ (interfaces are specified in api.py/api.h). The description document need to descript how to test.
    Metrics:
    (1) ROC and AUC.


Call for AI Challenge Sponsors


BenchCouncil provides the AI challenges tracks---using data sets and benchmarks from AIBench (TR, Bench18), the testbed for reproducing performance data, and the communication tool (Xinxiu—a dedicated communication tool for science and education). The sponsors are expected to provide financial support or donate hardware or software.