The Tutorial on BenchCouncil AIBench Scenario, Training, Inference, Micro and Synthetics Benchmarks across Datacenter, HPC, IoT, and Edge


Tutorial Zoom Link


International Open Benchmark Council (BenchCouncil) is a non-profit international benchmark organization, which aims to promote the standardization, benchmarking, evaluation, incubation, and promotion of open-source chip, AI, and Big Data techniques. This tutorial aims at presenting BenchCouncil AIBench---a comprehensive AI benchmark suite including AIBench Scenario, Training, Inference, Micro, and Synthetic Benchmarks across Datacenter, HPC, IoT and Edge. We also provide hands-on demos using AIBench on the BenchCouncil testbed---an open testbed for AI in HPC, Datacenter, IoT, and Edge. We want to explain the rationale and the methodology behind the benchmark design, plus a series of benchmarking tools, testbed, results, and performance rankings. We are glad to introduce the following interesting topics:
(1) AI benchmarking challenge.
(2) AIBench methodology & summary.
(3) AIBench Scenario: Scenario-distilling AI Benchmarking.
(4) Edge AIBench: towards Comprehensive End-to-end Edge Computing Benchmarking.
(5) AIBench training, subset, and its rankings.
(6) HPC AI500: A benchmark suite and HPC AI ranking for HPC AI systems.
(7) AIBench Inference benchmarks.
(8) AIoTBench for benchmarking mobile and embedded device Intelligence and its rankings.
(9) AIBench Micro and Synthetic Benchmarks.
(10) An open testbed for AI in HPC, Datacenter, IoT, and Edge.
(11) Hands-on demos on how to use AIBench.

Location and Date

We will give a tutorial on AIBench at ASPLOS 2021.

Time: Wednesday, April 14 | Full Day

Location: Zoom Link

Organizers and Presenters

Organizer: Prof. Jianfeng Zhan ICT, Chinese Academy of Sciences, and BenchCouncil
Dr. Wanling Gao ICT, Chinese Academy of Sciences, and BenchCouncil
Presenter: Prof. Jianfeng Zhan ICT, Chinese Academy of Sciences, and BenchCouncil
Dr. Lei Wang ICT, Chinese Academy of Sciences, and BenchCouncil
Dr. Wanling Gao ICT, Chinese Academy of Sciences, and BenchCouncil
Chunjie Luo ICT, Chinese Academy of Sciences, and University of Chinese Academy of Sciences
Chuanxin Lan ICT, Chinese Academy of Sciences
Tianshu Hao ICT, Chinese Academy of Sciences, and University of Chinese Academy of Sciences
Zihan Jiang ICT, Chinese Academy of Sciences, and University of Chinese Academy of Sciences
Fei Tang ICT, Chinese Academy of Sciences, and University of Chinese Academy of Sciences

Abstract

As a joint work with seventeen industry partners, AIBench is a comprehensive AI benchmark suite, distilling and abstracting real-world application scenarios into AI Scenario, Training, Inference, Micro and Synthetics Benchmarks across Datacenter, HPC, IoT, and Edge. AIBench Scenario benchmarks are proxies to industry-scale real-world applications scenarios. Each scenario benchmark models the critical paths of a real-world application scenario as a permutation of the AI and non-AI modules. Edge AIBench is an instance of the scenario benchmark suites, modeling end-to-end performance across IoT, edge, and Datacenter. AIBench Training and AIBench Inference cover nineteen representative AI tasks with state-of-the-art models to guarantee diversity and representativeness. AIBench Micro provides the intensively-used hotspot functions, profiled from the complete AIBench benchmarks, for simulation-based architecture researches. AIBench Synthetic is complementary to real-world benchmarks, with scalable problem sizes to model learning dynamics. AI training is prohibitively costly; AIBench Training provides two subsets for repeatable benchmarking and workload characterization to improve affordability; they keep the benchmarks to a minimum while maintaining representativeness. Based on the AIBench Training subset for repeatable benchmarking, we provide HPC AI500 to evaluate large-scale HPC AI systems. AIoTBench implements the AI inference benchmarks on various IoT and embedded devices, emphasizing diverse lightweight AI frameworks and models. Finally, the hands-on demos illustrate how to use AIBench on the BenchCouncil Testbed, which is publicly available.

Schedule

Wednesday, April 14 | 7am-11am PT

Time (am PT) Topic Speaker Slides
7:00-8:00 AI Benchmarking Challenges, AIBench Methodology and Summary Jianfeng Zhan
8:00-8:30 AIBench Scenario: Scenario-distilling AI Benchmarking Tianshu Hao
8:30-8:45 Hands-on demos on how to use AIBench Scenario Fei Tang
8:45-8:55 Break
8:55-9:25 Edge AIBench: towards Comprehensive End-to-end Edge Computing Benchmarking Tianshu Hao
9:25-9:40 Hands-on demos on how to use Edge AIBench Chuanxin Lan
9:40-10:10 AIBench training, subsets, and its rankings Fei Tang
10:10-10:25 Hands-on demos on how to use AIBench Training Chuanxin Lan
10:25-11:00 Discussion

Wednesday, April 14 | 4pm-8pm PT

Time (am PT) Topic Speaker Slides
4:00-4:30 HPC AI500: A Benchmark Suite and HPC AI Ranking for HPC AI Systems Zihan Jiang
4:30-4:45 Hands-on Demos on How to Use HPC AI500 Chuanxin Lan
4:45-5:05 AIBench Inference Benchmarks Chuanxin Lan
5:05-5:15 Hands-on Demos on How to Use AIBench Inference Chuanxin Lan
5:15-5:45 AIoTBench for Benchmarking Mobile and Embedded Device Intelligence and Its Rankings Chunjie Luo
5:45-6:00 Hands-on Demos on How to Use AIoTBench Chunjie Luo
6:00-6:10 Break
6:10-6:30 AIBench Micro and Synthetic Benchmarks Fei Tang
6:30-6:40 Hands-on Demos on How to Use AIBench Micro Fei Tang
6:40-8:00 Discussion

Publications

Project Homepage: https://www.benchcouncil.org/aibench.

* AIBench Training

AIBench Training: Balanced Industry-Standard AI Training Benchmarking.[PDF]

Fei Tang, Wanling Gao, Jianfeng Zhan, Chuanxin Lan, Xu Wen, Lei Wang, Chunjie Luo, Zheng Cao, Xiongwang Xiong, Zihan Jiang, Tianshu Hao, Fanda Fan, Fan Zhang, Yunyou Huang, Jianan Chen, Mengjia Du, Rui Ren, Chen Zheng, Daoyi Zheng, Haoning Tang, Kunlin Zhan, Biao Wang, Defei Kong, Minghe Yu, Chongkang Tan, Huan Li, Xinhui Tian, Yatao Li, Junchao Shao, Zhenyu Wang, Xiaoyu Wang, Jiahui Dai, Hainan Ye. 2021 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS 2021).

AIBench: Towards Scalable and Comprehensive Datacenter AI Benchmarking. [PDF]

Wanling Gao, Chunjie Luo, Lei Wang, Xingwang Xiong, Jianan Chen, Tianshu Hao, Zihan Jiang, Fanda Fan, Mengjia Du, Yunyou Huang, Fan Zhang, Xu Wen, Chen Zheng, Xiwen He,, Jiahui Dai, Hainan Ye, Zheng Cao, Zhen Jia, Kent Zhan, Haoning Tang, Daoyi Zheng, Biwei Xie, Wei Li, Xiaoyu Wang, and Jianfeng Zhan. 2018 BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing (Bench18).


* AIBench Scenario

AIBench Scenario: Scenario-distilling AI Benchmarking. [PDF]

Wanling Gao, Fei Tang, Jianfeng Zhan, Xu Wen, Lei Wang, Zheng Cao, Chuanxin Lan, Chunjie Luo, Xiaoli Liu, Zihan Jiang. arXiv:2005.03459v2.

AIBench: An Industry Standard Internet Service AI Benchmark Suite. [PDF]

Wanling Gao, Fei Tang, Lei Wang, Jianfeng Zhan, Chunxin Lan, Chunjie Luo, Yunyou Huang, Chen Zheng, Jiahui Dai, Zheng Cao, Daoyi Zheng, Haoning Tang, Kunlin Zhan, Biao Wang, Defei Kong, Tong WW u, Minghe Yu, Chongkang Tan, Huan Li, Xinhui Tian, Yatao Li, Gang Lu, Junchao Shao, Zhenyu Wang, Xiaoyu Wang, and Hainan Ye. Technical Report, 2019.


* Edge AIBench

Edge AIBench: Towards Comprehensive End-to-end Edge Computing Benchmarking. [PDF]

Tianshu Hao, Yunyou Huang, Xu Wen, Wanling Gao, Fan Zhang, Chen Zheng, Lei Wang, Hainan Ye, Kai Hwang, Zujie Ren, and Jianfeng Zhan. 2018 BenchCouncil Internationall Symposium on Benchmarking, Measuring and Optimizing (Bench18)


* HPC AI500

HPC AI500: Representative, Repeatable and Simple HPC AI Benchmarking. [PDF]

Zihan Jiang, Wanling Gao, Fei Tang, Xingwang Xiong, Lei Wang, Chuanxin Lan, Chunjie Luo, Hongxiao Li, Jianfeng Zhan. Revised Technical Report, 2021.

HPC AI500: The Methodology, Tools, Roofline Performance Models, and Metrics for Benchmarking HPC AI Systems. [PDF]

Zihan Jiang, Lei Wang, Xingwang Xiong, Wanling Gao, Chunjie Luo, Fei Tang, Chuanxin Lan, Hongxiao Li, Jianfeng Zhan. Technical Report, 2020.

HPC AI500: A Benchmark Suite for HPC AI Systems. [PDF]

Zihan Jiang, Wanling Gao, Lei Wang, Xingwang Xiong, Yuchen Zhang, Xu Wen, Chunjie Luo, Hainan Ye, Xiaoyi Lu, Yunquan Zhang, Shengzhong Feng, Kenli Li, Weijia Xu, and Jianfeng Zhan. 2018 BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing (Bench18).


* AIoTBench

Comparison and Benchmarking of AI Models and Frameworks on Mobile Devices. [PDF]

Chunjie Luo, Xiwen He, Jianfeng Zhan, Kexin Zhao, Lei Wang, Wanling Gao, Shaopeng Dai. Technical Report, 2020.

AIoTBench: Towards Comprehensive Benchmarking Mobile and Embedded device Intelligence. [PDF]

Chunjie Luo, Fan Zhang, Cheng Huang, Xingwang Xiong, Jianan Chen, Lei Wang, Wanling Gao, Hainan Ye, Tong Wu, Runsong Zhou, and Jianfeng Zhan. 2018 BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing (Bench18)

Biographies

Jianfeng Zhan
Prof. Jianfeng Zhan is a Full Professor at the Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS), and University of Chinese Academy of Sciences (UCAS), and director of the Software Systems Lab, ICT, CAS. He received his B.E. in Civil Engineering and MSc in Solid Mechanics from Southwest Jiaotong University in 1996 and 1999, and his Ph.D. in Computer Science from the Institute of Software, CAS, and UCAS in 2002. He has supervised over 90 graduate students, post-docs, and engineers in the past two decades. His research areas span from Chips, Systems to Benchmarks. A common thread is benchmarking, designing, and implementing, and optimizing parallel and distributing systems. He has made substantial and compelling efforts to transfer his academic research into advanced technology to impact general-purpose production systems. Several technical innovations and research results, including 35 patents, from his team, have been widely adopted in benchmarks, operating systems, and cluster and cloud system software with direct contributions to the advancement of the parallel and distributed systems in China or even in the world. Prof. Jianfeng Zhan founds and chairs BenchCouncil. He has served as IEEE TPDS Associate Editor since 2018. He received the second-class Chinese National Technology Promotion Prize in 2006, the Distinguished Achievement Award of the Chinese Academy of Sciences in 2005, and IISWC Best paper award in 2013, respectively.

Lei Wang
Dr. Lei Wang received the Ph. D degree in computer engineering from the University of Chinese Academy of Sciences, Beijing, China, in 2016. He is currently a senior engineer with the Institute of Computing Technology, Chinese Academy of Sciences. His current research interests include data center software systems, workload characterization, and benchmarking. He was a recipient of the Distinguished Achievement Award of the Chinese Academy of Sciences in 2005.

Wanling Gao
Dr. Wanling Gao is an Assistant Professor in computer science at the Institute of Computing Technology, Chinese Academy of Sciences, and University of Chinese Academy of Sciences. Her research interests focus on big data benchmarks and big data analytics. She received her B.S. degree in 2012 from Huazhong University of Science and Technology and her Ph.D. degree in 2019 from the Institute of Computing Technology, Chinese Academy of Sciences, and University of Chinese Academy of Sciences in China.

Chunjie Luo
Chunjie Luo is an Engineer and Ph.D. candidate of the Institute of Computing Technology, Chinese Academy of Sciences, and University of Chinese Academy of Sciences. His research interests focus on machine learning and benchmarking. He received his B.S. degree in 2009 from Huazhong University of Science and Technology and his master's degree in 2012 from the Institute of Computing Technology, Chinese Academy of Sciences, and University of Chinese Academy of Sciences in China.
Chuanxin Lan
Chuanxin Lan received the master's degree in the school of artificial intelligence from Hebei University of Technology, Tianjin, China, in 2019. He is now working as an assistant engineer in the Center for Advanced Computer Systems, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China. Besides, his research interests include artificial intelligence and benchmarking.
Tianshu Hao
Tianshu Hao received the B.S. degree from Nankai University, Tianjin, China, in 2015. She is currently pursuing Ph. D. degree in ICT, CAS. Her research interests focus on big data, edge computing, IoT, and AI benchmarking.

Zihan Jiang
Zihan Jiang is a doctoral student in computer architecture at the Institute of Computing Technology, Chinese Academy of Sciences, and the University of Chinese Academy of Sciences. His research interests include AI benchmark and distributed deep learning. Currently, he works on the HPC AI500 project.

Fei Tang
Fei Tang received the B.S. degree from Zhengzhou University, Zhengzhou, China, in 2016. He is currently pursuing Ph. D. degree in ICT, CAS. His research interests focus on big data, benchmarking, and search engines.