Challenges 2019
"2019 BenchCouncil International Artificial Intelligence System Challenges" consists of system challenges (on many kinds of chips such as Cambricon, X86, RISC-V), and Intellifusion algorithm challenge.
The result submission extented to Oct 7.
Submission site: http://125.39.136.212:8090/
More details about AI Challenges are available from the challenge manual: http://www.benchcouncil.org/competition/handbook-en.pdf.
Awards
500,000 CNY in total, about 70K US Dollar.
- Special Award (Only one): 100,000 CNY
- The First Prize for Every Track: 30,000 CNY
- The Second Prize for Every Track: 20,000 CNY
- The Third Prize for Every Track: 10,000 CNY
2019 Challenge Papers
- International AI System Challenge based on RISC-V
- RVTensor: A light-weight neural network inference framework based on the RISC-V architecture [PDF]
Four Challenges Tracks
The topics of four tracks are derived from AIBench (TR, Bench18). Also, the provided models are trained using AIBench benchmarks.
Download AIBench (Hosted on BenchCouncil code management system---BenchHub):
Micro Benchmarks: http://125.39.136.212:8090/AIBench/DC_AIBench_Micro
Component Benchmarks: http://125.39.136.212:8090/AIBench/DC_AIBench_Component
Application Benchmarks: http://125.39.136.212:8090/AIBench/AIBench_DCMIX
- Track 1：International AI System Challenge based on RISC-V Subject：
- Track 2：International AI System Challenge based on Cambricon Chip Subject：
- Track 3：International AI System Challenge based on X86 Platform Subject：
- Track 4：International 3D Face Recognition Algorithm Challenge Subject：
The implementation and optimization of convolutional neural network based image classification task on RISC-V.
Requirements：
(1) Implement the forward Calculation of neural network. Load and run the given trained model ResNet-20, the format of model is hdf5.
Download simulator：https://hub.docker.com/r/crva/riscv-qemu
Download data：https://www.cs.toronto.edu/~kriz/cifar-10-binary.tar.gz
Download model： http://www.benchcouncil.org/competition/cifar_resnet.zip
(2) Minimize external dependences, and try not to use external libraries (e.g., OpenMP, Pthread, Boost).
(3) Use the model required by the organizer and guarantee the original model accuracy (84.03%). The accuracy deviation with the result of organizer provided model is within 0.05%.
Metrics：
(1) Minimize the binary file (e.g., the executable files compiled by RISC-V compiler, model weights, etc).
(2) Maximize the execution performance and minimize the number of instructions.
2019 AI Challenge Awards Recipients:
Jiageng Yu is a Senior Engineer at the Intelligent Software Research Center(ISRC)
of Institute of Software Chinese Academy of Sciences(ISCAS), and received his
Ph.D. from the Graduate School of the Chinese Academy of Sciences in 2012. His
main research focuses on intelligent system software, operating system, cloud
computing, etc.
Yuxia Miao is now a R&D Engineer at the Intelligent Software Research Center(ISRC) of Institute
of Software Chinese Academy of Sciences(ISCAS), and received her master's degree in the school
of information science and engineering at the Yanshan University in 2017. Her research interests
lie in the areas of machine learning, image processing (classification, detection and recognition).
Yang Tai is currently working as a R&D Engineer at the Intelligent Software Research
Center(ISRC) of Institute of Software Chinese Academy of Sciences(ISCAS), and received
his master's degree in computer science and technology from the Institute of Telecommunications
in southern Paris, France, in 2017. He is mainly engaged in the research of the operating systems
at intelligent server based on container technology, and the optimization and adaptation of AI
inference in smart chips based in RISC-V architecture.
Contribution: We firstly achieves efficient compilation and deployment for SERVE.r board by using the Freedom Open
Source SDK, and secondly minimizes the third-party dependencies by writing modules for the basic
neural network data structures, operators, etc. It is only 193KB of executable files of RVTensor
and the execution speed is expected. Thirdly, the accuracy of the inference is consistent with Keras
by implementing the analysis of the H5 model and the network execution module.
Yangyang Kong is a research assistant at the State Key Laboratory of Information Security,
Institute of Information Engineering, Chinese Academy of Sciences. He received an ME in computer
technology from the University of Chinese Academy of Sciences. His main research interests include
computer architecture and artificial intelligence. Contact him at kongyangyang@iie.ac.cn.
Contribution: Explore approaches to implementing deep learning inference on RSIC-V platforms,
and evaluate the performance of running deep learning inference applications on multiple systems.
The implementation and optimization of convolutional neural network based image classification task on Cambricon.
Requirements：
(1) Implement the forward Calculation of neural network. Load and run the given trained model ResNet-50.
Download data (image)：http://125.39.136.212:8484/Cambricon/cifar10_test.tar
Download data (binary)：https://www.cs.toronto.edu/~kriz/cifar-10-binary.tar.gz
Download model： http://125.39.136.212:8484/Cambricon/model.tar
(2) Use the model required by the organizer and guarantee the original model accuracy (84.39%). The accuracy deviation with the result of organizer provided model is within 0.05%.
(3) The result submission should include source code, description document, and model accuracy.
Metrics：
(1) Maximize the execution performance and minimize the prediction time (wall clock time) on provided test data.
2019 AI Challenge Awards Recipients:
Advisor :Prof. Xiaobing Feng Professor at Institute of Computing Technology, Chinese Academy of Sciences.
Guangli Li is a Ph.D. student at Institute of Computing Technology, Chinese Academy of Sciences.
Xueying Wang is a Ph.D. student at Institute of Computing Technology, Chinese Academy of Sciences.
Xiu Ma is a Ph.D. student at Jilin University and a visiting student at ICT, CAS.
Contribution: We proposed a fusion-based optimization methodology by exploring the characteristics
of Cambricon chips and designed an efficient optimization and inference engine, achieving 7.44x speedup.
Zihan Jiang is a Ph.D. candidate in computer architecture of University of Chinese Academy
of Sciences (UCAS). His research interests includes AI benchmarking, performance analysis and distributed deep learning.
Jiansong Li is a Ph.D. candidate of University of Chinese Academy of Sciences (UCAS). His research interests span
compiler techniques, runtime system, programming language, computer architecture and etc.
Contribution: We quantitative analysis the impact of frequent-used inference optimizations on accuracy and throughput
on Cambricon MLU100 platforms, and we achieve the CIFAR10 inference in milliseconds with the guarantee of accuracy.
Yifan Wang, Ph.D candidate at Institute of Computing Technology, Chinese Academy of Science. His main research interests
include computer architecture, edge computing, and system performance modeling and evaluation.
Chen Zeng, Ph.D candidate at Institute of Computing Technology, Chinese Academy of Science. Her main research interests
include distributed computing and blockchain.
Chundian Li, Ph.D candidate at Institute of Computing Technology, Chinese Academy of Science. His main research interests
include distributed storage, cloud computing, and micro-architecture.
Contribution: We leverage offline model, multiple threads programming, and I/O data structure optimization to explore the
performance bound of Cambricon accelerator MLU100 in end-to-end deep learning inference scenarios. The evaluation result shows,
for a specific inference workload, RetNet-50 on CIFAR-10 dataset, our optimization methods achieve 15, 054 FPS without changing
the model structure.
Advisor : Ping Yao received the B. E. degree in Automation in Tsinghua University in 1996,
and the Ph.D degree in Communication and Information System from Institute of Electronics,
Chinese Academy of Sciences in 2003 respectively. She is now an associate researcher in
Institute of Computing Technology, Chinese Academy of Sciences. Her research interests
include digital signal and image processing, remote sensing data processing and
application.
Peng He, a postgraduate student from the Institute of Computing Technology,
Chinese Academy of Sciences, is engaged in research on digital image processing.
Ge Chen, a postgraduate student from the Institute of Computing Technology,
Chinese Academy of Sciences, is engaged in research on digital image processing.
Kai Deng, a postgraduate student from the Institute of Computing Technology,
Chinese Academy of Sciences, is engaged in research on digital image processing.
Contribution: We deploy the resnet101 classification model on the Cambricon platform. In order to get the best performance,
we conduct several experiments to determine the proper degree of the model parallelism, data parallelism and the number of threads.
The implementation and optimization of matrix decomposition based collaborative filtering task on X86 platform.
Requirements：
(1) Implement ALS-WR training algorithm on X86 platform. The competitors can use external libraries supported by the platform.
Download data：https://grouplens.org/datasets/movielens/
(2) The parameter nf chooses 100. Training 30 rounds using Movielens dataset, shorter training time is better.
Metrics：
(1) Maximize the execution performance and minimize the training time (wall clock time) on provided data.
2019 AI Challenge Awards Recipients:
Bio: We are from Sustainable Architectures and Infrastructure Laboratory, Shanghai Jiao Tong University.
Contribution: We exploit parallelism, sparsity and locality to accelerate matrix factorization on x86 platforms.
Tianshu Hao received the B.S. degree from Nankai University, Tianjin, China, in 2015. She is currently pursuing
Ph. D. dergree in ICT, CAS. Her research interests focus on big data, edge computing, IoT and AI benchmarking.
Ziping Zheng received M.S. degree from School of Computer Science, Carnegie Mellon University, in 2016. He's
currently a software engineer working at Google. He's watching closely the cutting-edge technology in the area of
machine learning and self-less driving.
Contribution: They propose an automatic granularity tuning framewrok, which can efficiently resolve the load
imbalance of threads in the ALS-WR algorithm.
Maosen Chen graduated from the Institute of Computing Technology of Chinese Academy of Sciences. His research
direction is the recommendation system and NLP. He is currently a senior algorithm engineer in the 360 Information
Stream Products Department, responsible for NLP and recall related work.
Qianyun Chen is currently an undergraduate at the Georgia Institute of Technology. Her research interests
include heterogeneous computing, code optimization, and high performance computing.
Tun Chen received the master degree in computer science from Hunan Normal University in 2018. He is currently
working toward the Ph.D. degree at the State Key Laboratory of Computer Architecture, Institute of Computing Technology,
Chinese Academy of Sciences. His research interests include high performance computing, heterogeneous computing, in-core
parallelism, and optimized FFT library.
Yi Liang received the Ph.D. degree in Institute of Computing Technology, Chinese Academy of Sciences, in 2006.
She is currently an Associate Professor with the Beijing University of Technology. Her research interests include
Bigdata system, computer architecture and cloud computing.
Shaokang Zeng received the B.S. degree in computer science and technology from Beijing University of Technology,
China, in 2018, where he is currently pursuing the M.S. degree in computer science and technology. His research interests
include Bigdata system, computer architecture and cloud computing.
Yande Liang received the B.S. degree in computer science and technology from Shandong University of Science and
Technology, China, in 2018, and he is currently pursuing the M.S. degree in computer science and technology in Beijing
University of Technology. His research interests include Bigdata system, computer architecture and cloud computing.
Kaizhong Chen received the B.S. degree in computer science and technology from Beijing University of Technology,
China, in 2019, where he is currently pursuing the M.S. degree in computer science and technology. His research interests
include Bigdata system, computer architecture and cloud computing.
Contribution: Accelerating Parallel ALS for Collaborative Filtering on Hadoop
In our work, we propose an integrated optimized solution for parallel ALS on Hadoop, which incorporates the record-based
rating data partitioning, multithread-based fine-grained parallelism within the map task and the optimization of JVM heap
size configuration. Experimental results demonstrate that our solution can reduce the execution time of Hadoop ALS by 98.95%
by maximum, while not hurting RMSE of Hadoop ALS significantly.
3DFRC (3D Face Recognition Challenge) aims at soliciting new approaches to advance the state-of-the-art in face recognition.
Requirements：
(1) The competitors are optional to use external data for model training.
Download data：http://125.39.136.212:8484/3dvggface2_1.tar.gz
(2) The external data used for training must be descripted in "method description" file.
(3) The competitors need to submit the model file and test file.
(4) Source code for test and description document should be submitted. The source code for test need to implement related interfaces using python3 and c++ (interfaces are specified in api.py/api.h). The description document need to descript how to test.
Metrics：
(1) ROC and AUC.
2019 AI Challenge Awards Recipients:
Xingwang Xiong is a second-year graduate student of Institute of Computing Technology (ICT) of the Chinese Academy
of Sciences (CAS). He received his B.S. degree in Computer Science in Changsha University of Sciences and Technology in
2018. His research interests include Distributed Machine Learning and Benchmarking.
Xu Wen is a master student of Institute of Computing Technology (ICT) of the Chinese Academy of Sciences (CAS).
He received his B.S. degree from University of Chinese Academy of Sciences, Beijing, China, in 2018. His research interests
include Distributed Machine Learning and Benchmarking.
Cheng Huang received the B.S. degree from University of Chinese Academy of Sciences, Beijing, China, in 2018. He is currently
pursuing M.S. degree in ICT, CAS. His research interests include big data and computer system.
Contribution: Traditional 2D face recognition is sensitive to variations of the external environment, while 3D face data is
insufficient to train an efficient model. To this end, we used inter-modal transfer learning and achieved an accuracy of 94.64%
on IntelliFusion 3D face dataset.
Tongyan Gong received B.S. degree from the College of Technology and Engineering,
Lanzhou University of Technology, in 2016. She is currently a joint training master degree
candidate of Guizhou university of finance and economics and the Institute of Computing
Technology, Chinese Academy of Sciences. Her research interests include parallel processing
and high-performance computing.
Huiqian Niu received her master degree from the College Computer Science, Nankai University, in 2018. She is currently
a data mining engineer working at Technology and data center of JD. Her research interests include image processing,
big data and machine learning.
Contribution: They implement ResNet model on RGB-D facial recognition dataset, analyzing the dataset and optimizing the perfomance.
Heming Sun from Jiangxi China, junior student at the Ohio State University.
Xi Xiong from Beijing China, junior student at the Ohio State University.
Contribution: The team solved the 3D facial recognition problem with a modified version of Resnet-18 on pytorch framework.
It reached 86% validation precision.
Call for AI Challenge Sponsors
BenchCouncil provides the AI challenges tracks---using data sets and benchmarks from AIBench (TR, Bench18), the testbed for reproducing performance data, and the communication tool (Xinxiu—a dedicated communication tool for science and education). The sponsors are expected to provide financial support or donate hardware or software.