Tutorial: BenchCouncil AIBench

---Distilling Real-world applications into AI Scenario, Training, and Inference Benchmarks across Datacenter, HPC, IoT and Edge


International Open Benchmark Council (BenchCouncil) is a non-profit international benchmark organization, which aims to promote the standardization, benchmarking, evaluation, incubation, and promotion of open-source chip, AI, and Big Data techniques. This tutorial aims at presenting BenchCouncil AIBench---a comprehensive AI benchmark suite for Datacenter, HPC, IoT and Edge. We also provide hands-on demos on how to use AIBench on the BenchCouncil testbed---an open testbed for AI in HPC, Datacenter, IoT, and Edge. We would like to explain the rationale and the methodology behind the benchmark design, plus a series benchmarking tools, testbed, results, and performance rankings. We are glad to introduce the following interesting topics:
(1) The challenges and motivation for characterizing AI workloads.
(2) Benchmarking methodology, models, and metrics.
(3) AIBench scenario benchmarks.
(4) Edge AIBench: towards Comprehensive End-to-end Edge Computing Benchmarking.
(5) AIBench training and its rankings.
(6) HPC AI500: A benchmark suite and HPC AI ranking for HPC AI systems.
(7) AIBench Inference and its rankings.
(8) AIoTBench for benchmarking mobile and embedded device Intelligence and its rankings.
(9) An open testbed for AI in HPC, Datacenter, IoT, and Edge.
(10) Hands-on demos on how to use AIBench on BenchCouncil testbed.

Location and Date

We will give a tutorial on AIBench at ISCA 2021.

Time: TBD

Location: TBD

Organizers and Presenters

Organizer: Prof. Jianfeng Zhan ICT, Chinese Academy of Sciences, and BenchCouncil
Dr. Wanling Gao ICT, Chinese Academy of Sciences, and BenchCouncil
Presenter: Prof. Jianfeng Zhan ICT, Chinese Academy of Sciences, and BenchCouncil
Dr. Lei Wang ICT, Chinese Academy of Sciences, and BenchCouncil
Dr. Wanling Gao ICT, Chinese Academy of Sciences, and BenchCouncil
Chunjie Luo ICT, Chinese Academy of Sciences, and University of Chinese Academy of Sciences
Tianshu Hao ICT, Chinese Academy of Sciences, and University of Chinese Academy of Sciences
Zihan Jiang ICT, Chinese Academy of Sciences, and University of Chinese Academy of Sciences
Fei Tang ICT, Chinese Academy of Sciences, and University of Chinese Academy of Sciences

Abstract

Modern real-world application scenarios like Internet services not only consist of diversity of AI and non-AI modules with very long and complex execution paths, but also have huge code size, which raises serious benchmarking or evaluating challenges. Using AI components or micro benchmarks alone can lead to error-prone conclusions. Together with seventeen industry partners, we extract nine typical application scenarios, and identify the primary components. As the proxy to real-world applications, the AIBench scenario benchmarks let the software and hardware designers obtain the overall system performance and find out the key components within the critical path. Following the same methodology, we propose Edge AIBench for benchmarking end-to-end performance across IoT, edge and Datacenter.

Earlier-stage evaluations of a new AI architecture/system need affordable AI training benchmarks, while using a few AI component benchmarks alone in the other stages may lead to misleading conclusions. We present a balanced AI benchmarking methodology for meeting the conflicting requirements of different stages. We identify and implement seventeen representative AI tasks with the state-of-the-art models to guarantee the diversity and representativeness of the benchmarks. Meanwhile, we keep a benchmark subset to a minimum for affordability. Furthermore, on the basis of the AIBench training subset, we present the HPC AI500 benchmarks for evaluating HPC AI systems for both affordability and representativeness. For AI Inference, as its cost is trivial, we provide comprehensive AI inference benchmarks. Meanwhile, we propose AIoTBench for considering diverse light-weight AI frameworks and models.

Schedule

TBD

Publications

Scenario-distilling AI Benchmarking. [PDF]
Wanling Gao, Fei Tang, Jianfeng Zhan, Xu Wen, Lei Wang, Zheng Cao, Chuanxin Lan, Chunjie Luo, Xiaoli Liu, Zihan Jiang. arXiv:2005.03459v2.

AIBench: An Agile Domain-specific Benchmarking Methodology and an AI Benchmark Suite. [PDF]
Wanling Gao, Fei Tang, Jianfeng Zhan, Chuanxin Lan, Chunjie Luo, Lei Wang, Jiahui Dai, Zheng Cao, Xiongwang Xiong, Zihan Jiang, Tianshu Hao, Fanda Fan, Xu Wen, Fan Zhang, Yunyou Huang, Jianan Chen, Mengjia Du, Rui Ren, Chen Zheng, Daoyi Zheng, Haoning Tang, Kunlin Zhan, Biao Wang, Defei Kong, Minghe Yu, Chongkang Tan, Huan Li, Xinhui Tian, Yatao Li, Gang Lu, Junchao Shao, Zhenyu Wang, Xiaoyu Wang, Hainan Ye. Technical Report, 2020.

BenchCouncil’s View On Benchmarking AI and Other Emerging Workloads. [PDF]
Jianfeng Zhan, Lei Wang, Wanling Gao, and Rui Ren. Technical Report, 2019.

AIBench: An Industry Standard Internet Service AI Benchmark Suite. [PDF]
Wanling Gao, Fei Tang, Lei Wang, Jianfeng Zhan, Chunxin Lan, Chunjie Luo, Yunyou Huang, Chen Zheng, Jiahui Dai, Zheng Cao, Daoyi Zheng, Haoning Tang, Kunlin Zhan, Biao Wang, Defei Kong, Tong Wu, Minghe Yu, Chongkang Tan, Huan Li, Xinhui Tian, Yatao Li, Gang Lu, Junchao Shao, Zhenyu Wang, Xiaoyu Wang, and Hainan Ye. Technical Report, 2019.

AIBench: Towards Scalable and Comprehensive Datacenter AI Benchmarking. [PDF]
Wanling Gao, Chunjie Luo, Lei Wang, Xingwang Xiong, Jianan Chen, Tianshu Hao, Zihan Jiang, Fanda Fan, Mengjia Du, Yunyou Huang, Fan Zhang, Xu Wen, Chen Zheng, Xiwen He, Jiahui Dai, Hainan Ye, Zheng Cao, Zhen Jia, Kent Zhan, Haoning Tang, Daoyi Zheng, Biwei Xie, Wei Li, Xiaoyu Wang, and Jianfeng Zhan. 2018 BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing (Bench18).

HPC AI500: A Benchmark Suite for HPC AI Systems. [PDF]
Zihan Jiang, Wanling Gao, Lei Wang, Xingwang Xiong, Yuchen Zhang, Xu Wen, Chunjie Luo, Hainan Ye, Xiaoyi Lu, Yunquan Zhang, Shengzhong Feng, Kenli Li, Weijia Xu, and Jianfeng Zhan. 2018 BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing (Bench18).

AIoTBench: Towards Comprehensive Benchmarking Mobile and Embedded device Intelligence. [PDF]
Chunjie Luo, Fan Zhang, Cheng Huang, Xingwang Xiong, Jianan Chen, Lei Wang, Wanling Gao, Hainan Ye, Tong Wu, Runsong Zhou, and Jianfeng Zhan. 2018 BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing (Bench18).

Edge AIBench: Towards Comprehensive End-to-end Edge Computing Benchmarking. [PDF]
Tianshu Hao, Yunyou Huang, Xu Wen, Wanling Gao, Fan Zhang, Chen Zheng, Lei Wang, Hainan Ye, Kai Hwang, Zujie Ren, and Jianfeng Zhan. 2018 BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing (Bench18).

DCMix: Generating mixed workloads for the cloud data center. [PDF]
Xingwang Xiong, Lei Wang, Wanling Gao, Rui Ren, Ke Liu, Chen Zheng, Yu Wen, and Yi Liang. 2018 BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing (Bench18).

Data Motifs: A Lens Towards Fully Understanding Big Data and AI Workloads. [PDF]
Wanling Gao, Jianfeng Zhan, Lei Wang, Chunjie Luo, Daoyi Zheng, Fei Tang, Biwei Xie, Chen Zheng, Xu Wen, Xiwen He, Hainan Ye and Rui Ren. The 27th International Conference on Parallel Architectures and Compilation Techniques (PACT 2018).

BigDataBench: a Big Data Benchmark Suite from Internet Services. [PDF]
Lei Wang, Jianfeng Zhan, Chunjie Luo, Yuqing Zhu, Qiang Yang, Yongqiang He, WanlingGao, Zhen Jia, Yingjie Shi, Shujie Zhang, Cheng Zhen, Gang Lu, Kent Zhan, Xiaona Li, and Bizhu Qiu. The 20th IEEE International Symposium On High Performance Computer Architecture (HPCA-2014), February 15-19, 2014, Orlando, Florida, USA.

Data Motif-based Proxy Benchmarks for Big Data and AI Workloads. [PDF]
Wanling Gao, Jianfeng Zhan, Lei Wang, Chunjie Luo, Zhen Jia, Daoyi Zheng, Chen Zheng, Xiwen He, Hainan Ye, Haibin Wang, and Rui Ren. 2018 IEEE International Symposium on Workload Characterization (IISWC 2018).

BigDataBench: a Scalable and Unified Big Data and AI Benchmark Suite. [PDF]
Wanling Gao, Jianfeng Zhan, Lei Wang, Chunjie Luo, Daoyi Zheng, Rui Ren, Chen Zheng, Gang Lu, Jingwei Li, Zheng Cao, Shujie Zhang, and Haoning Tang. Technical Report, arXiv preprint arXiv:1802.08254, January 27, 2018.

BOPS, Not FLOPS! A New Metric and Roofline Performance Model For Datacenter Computing. [PDF]
Lei Wang, Jianfeng Zhan, Wanling Gao, Zihan Jiang, Rui Ren, Xiwen He, Chunjie Luo, Gang Lu, Jingwei Li. Technical Report, arXiv preprint arXiv:1801.09212, May 3, 2018.

Understanding Big Data Analytics Workloads on Modern Processors. [PDF]
Zhen Jia, Jianfeng Zhan, Lei Wang, Chunjie Luo, Wanling Gao, Yi Jin, Rui Han and Lixin Zhang. IEEE Transactions on Parallel and Distributed Systems, 28(6), 1797-1810, 2017.

Understanding Processors Design Decisions for Data Analytics in Homogeneous Data Centers. [PDF]
Zhen Jia, Wanling Gao, Yingjie Shi, Sally A. McKee, Jianfeng Zhan, Lei Wang, Lixin Zhang. IEEE Transactions on Big Data, 2017.

A Dwarf-based Scalable Big Data Benchmarking Methodology. [PDF]
Wanling Gao, Lei Wang, Jianfeng Zhan, Chunjie Luo, Daoyi Zheng, Zhen Jia, Biwei Xie, Chen Zheng, Qiang Yang, and Haibin Wang. arXiv preprint arXiv: 1711.03229

Characterizing data analysis workloads in data centers. [PDF]
Zhen Jia, Lei Wang, Jianfeng Zhan, Lixin Zhang, Chunjie Luo. 2013 IEEE International Symposium on Workload Characterization (IISWC 2013) (Best paper award).

Characterizing and Subsetting Big Data Workloads.[PDF]
Zhen Jia, Lei Wang, Jianfeng Zhan, Lixin Zhang, Chunjie Luo, Ninghui Sun. 2014 IEEE International Symposium on Workload Characterization (IISWC 2014)

Identifying Dwarfs Workloads in Big Data Analytics.  [PDF]
W Gao, C Luo, J Zhan, H Ye, X He, L Wang, Y Zhu, X Tian. 
arXiv preprint arXiv:1505.06872

BDGS: A Scalable Big Data Generator Suite in Big Data Benchmarking. [PDF]
Zijian Ming, Chunjie Luo, Wanling Gao, Rui Han, Qiang Yang, Lei Wang, and Jianfeng Zhan. In Advancing Big Data Benchmarks (pp. 138-154). Springer International Publishing.

Biographies

Jianfeng Zhan
Prof. Jianfeng Zhan is a Full Professor at Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS), and University of Chinese Academy of Sciences (UCAS), and director of the Software Systems Labs, ICT, CAS. He received his B.E. in Civil Engineering and MSc in Solid Mechanics from Southwest Jiaotong University in 1996, and 1999, and his Ph.D. in Computer Science from Institute of Software, CAS and UCAS in 2002. He has supervised over 90 graduate students, post-docs, and engineers in the past two decades. His research areas span from Chips, Systems, to Benchmarks. A common thread is benchmarking, designing, and implementing, and optimizing parallel and distributing systems. He has made strong and effective efforts to transfer his academic research into advanced technology to impact general-purpose production systems. Several technical innovations and research results, including 36 patents, from his team have been widely adopted in benchmarks, operating systems and cluster and cloud system software with direct contributions to the advancement of the parallel and distributed systems in China or even in the world. Prof. Jianfeng Zhan founds and chairs BenchCouncil. He served as IEEE TPDS Associate Editor since 2018. He received the second-class Chinese National Technology Promotion Prize in 2006, the Distinguished Achievement Award of the Chinese Academy of Sciences in 2005, and IISWC Best paper award in 2013, respectively.

Lei Wang
Dr. Lei Wang received the Ph. D degree in computer engineering from University of Chinese Academy of Sciences, Beijing, China, in 2016. He is currently a senior engineer with the Institute of Computing Technology, Chinese Academy of Sciences. His current research interests include datacenter software systems, workload characterization and benchmarking. He was a recipient of the Distinguished Achievement Award of the Chinese Academy of Sciences in 2005.
Wanling Gao
Dr. Wanling Gao is an Assistant Professor in computer science at the Institute of Computing Technology, Chinese Academy of Sciences and University of Chinese Academy of Sciences. Her research interests focus on big data benchmark and big data analytics. She received her B.S. degree in 2012 from Huazhong University of Science and Technology and her PhD degree in 2019 from Institute of Computing Technology, Chinese Academy of Sciences and University of Chinese Academy of Sciences in China.

Chunjie Luo
Chunjie Luo is an Engineer and PHD candidate of the Institute of Computing Technology, Chinese Academy of Sciences and University of Chinese Academy of Sciences. His research interests focus on machine learning and benchmarking. He received his B.S. degree in 2009 from Huazhong University of Science and Technology and his master degree in 2012 from Institute of Computing Technology, Chinese Academy of Sciences and University of Chinese Academy of Sciences in China.
Tianshu Hao
Tianshu Hao received the B.S. degree from Nankai University, Tianjin, China, in 2015. She is currently pursuing Ph. D. dergree in ICT, CAS. Her research interests focus on big data, edge computing, IoT and AI benchmarking.

Zihan Jiang
Zihan Jiang is a doctoral student in computer architecture at Institute of Computing Technology, Chinese Academy of Sciences and University of Chinese Academy of Sciences. His research interests includes AI benchmark and distributed deep learning. Currently, he works on HPC AI500 project.

Fei Tang
Fei Tang received the B.S. degree from Zhengzhou University, Zhengzhou, China, in 2016. He is currently pursuing Ph. D. dergree in ICT, CAS. His research interests focus on big data, benchmarking and search engine.