Tutorial: BenchCouncil AIBench

--- an AI (Artificial Intelligence) Benchmark suite for Datacenter, IoT, Edge, and HPC

International Open Benchmark Council (BenchCouncil) is a non-profit research institute which aims to promote the standardization, benchmarking, evaluation, incubation, and promotion of open-source chip, AI, and Big Data techniques. This tutorial is aimed at presenting AIBench---an artificial intelligence (in short, AI) benchmark suite. We are glad to introduce the following interesting topics:

(1) BenchCouncil: Present and Future.

(2) Sixteen prominent AI problem domains and the essentials of modern AI workloads.

(3) AIBench framework and micro, component, and end-to-end benchmarks.

(4) HPC AI500: A Benchmark Suite for HPC AI Systems.

(5) AIoT Bench: Towards Comprehensive Benchmarking Mobile and Embedded device Intelligence.

(6) Edge AIBench: Towards Comprehensive End-to-end Edge Computing Benchmarking.

(7) How to use AIBench.

(8) How to contribute to AIBench? How to join BenchCouncil (http://www.benchcouncil.org).

Location and Date

We will give a tutorial on AIBench at HPCA 2020 in San Diego, CA, USA.

Time: Feb 22-23, 2020

ROOM: TBD

Organizers and Presenters

Organizer: Dr. Jianfeng Zhan ICT, Chinese Academy of Sciences, and University of Chinese Academy of Sciences
Presenter: Dr. Jianfeng Zhan ICT, Chinese Academy of Sciences, and University of Chinese Academy of Sciences
Presenter: Chunjie Luo ICT, Chinese Academy of Sciences, and University of Chinese Academy of Sciences
Presenter: Dr. Wanling Gao ICT, Chinese Academy of Sciences, and University of Chinese Academy of Sciences
Presenter: Tianshu Hao ICT, Chinese Academy of Sciences, and University of Chinese Academy of Sciences
Presenter: Zihan Jiang ICT, Chinese Academy of Sciences, and University of Chinese Academy of Sciences
Presenter: Fei Tang ICT, Chinese Academy of Sciences, and University of Chinese Academy of Sciences

Abstract

As a multi-discipline, i.e., system, architecture, data management and machine learning, research and engineering effort from both industry and academia, BenchCouncil AI Benchmarks is a comprehensive AI benchmark suite for HPC, Datacenter, Edge, and IoT.

AIBench is the first industry scale AI benchmark suite, joint with seventeen industry partners. First, we present a highly extensible, configurable, and flexible benchmark framework, containing multiple loosely coupled modules like data input, prominent AI problem domains, online inference, offline training and automatic deployment tool modules. We analyze typical AI application scenarios from three most important Internet services domains, and then we abstract and identify sixteen prominent AI problem domains, including classification, image generation, text-to-text translation, image-to-text, image-to- image, speech-to-text, face embedding, 3D face recognition, object detection, video prediction, image compression, recommendation, 3D object reconstruction, text summarization, spatial transformer, and learning to rank. We implement sixteen component benchmarks for those AI problem domains, and further profile and implement twelve fundamental units of computation across different component benchmarks as the micro benchmarks.

HPC AI500 is a benchmark suite for evaluating HPC systems that running scientific DL workloads. Each workload from HPC AI500 bases on real scientific DL applications and covers the most representative scientific fields, namely climate analysis, cosmology, high energy physics, gravitational wave physics and computational biology. Currently, we choose 18 scientific DL benchmarks (For details, see Specification) from application scenarios, datasets, and software stack. Furthermore, we propose a set of metrics of comprehensively evaluating the HPC systems, considering both accuracy, performance as well as power and cost. In addition, we provide a scalable reference implementation of HPC AI500.

Edge AIBench is a benchmark suite for end-to-end edge computing including four typical application scenarios: ICU Patient Monitor, Surveillance Camera, Smart Home, and Autonomous Vehicle, which consider the complexity of all edge computing AI scenarios. In addition, Edge AIBench provides an end-to-end application benchmarking framework, including train, validate and inference stages. Table 1 shows the component benchmarks of Edge AIBench. Edge AIBench provides an end-to-end application benchmarking, consisting of train, inference, data collection and other parts using a general three-layer edge computing framework.

AIoT Bench is a comprehensive benchmark suite to evaluate the AI ability of mobile and embedded devices. Our benchmark 1) covers different application domains, e.g. image recognition, speech recognition and natural language processing; 2) covers different platforms, including Android devices and Raspberry Pi; 3) covers different development tools, including TensorFlow and Caffe2; 4) offers both end-to-end application workloads and micro workloads.

Schedule

Time Agenda Presenter Resources
08:30-09:00 BenchCouncil: Present and Future. Jianfeng Zhan [Slides]
09:00-09:30 Benchmarking Methodology, models, and metrics Jianfeng Zhan [Slides]
09:30-10:00 Summary of Benchmarks Jianfeng Zhan [Slides]
10:00-10:20 Coffee Break
10:20-11:10 Sixteen prominent AI problem domains and the essentials of modern AI workloads Fei Tang [Slides]
11:10-12:00 AIBench: An Industry Standard Internet Service AI Benchmark Suite Wanling Gao [Slides]
12:00-14:00 Lunch
14:00-14:50 HPC AI500: A Benchmark Suite for HPC AI Systems Zihan Jiang [Slides]
14:50-15:40 AIoT Bench: Towards Comprehensive Benchmarking Mobile and Embedded device Intelligence Chunjie Luo [Slides]
15:40-16:00 Coffee Break
16:00-16:50 Edge AIBench: Towards Comprehensive End-to-end Edge Computing Benchmarking Tianshu Hao [Slides]
16:50-17:50 How to use AIBench Wanling Gao, Chunjie Luo, Fei Tang, Tianshu Hao, Zihan Jiang [Slides]
17:50-18:00 How to contribute to AIBench? How to join BenchCouncil (http://www.benchcouncil.org) Jianfeng Zhan [Slides]

Publications

AIBench: An Industry Standard Internet Service AI Benchmark Suite. [PDF]
Wanling Gao, Fei Tang, Lei Wang, Jianfeng Zhan, Chunxin Lan, Chunjie Luo, Yunyou Huang, Chen Zheng, Jiahui Dai, Zheng Cao, Daoyi Zheng, Haoning Tang, Kunlin Zhan, Biao Wang, Defei Kong, Tong Wu, Minghe Yu, Chongkang Tan, Huan Li, Xinhui Tian, Yatao Li, Gang Lu, Junchao Shao, Zhenyu Wang, Xiaoyu Wang, and Hainan Ye. Technical Report, 2019.

AIBench: Towards Scalable and Comprehensive Datacenter AI Benchmarking. [PDF]
Wanling Gao, Chunjie Luo, Lei Wang, Xingwang Xiong, Jianan Chen, Tianshu Hao, Zihan Jiang, Fanda Fan, Mengjia Du, Yunyou Huang, Fan Zhang, Xu Wen, Chen Zheng, Xiwen He, Jiahui Dai, Hainan Ye, Zheng Cao, Zhen Jia, Kent Zhan, Haoning Tang, Daoyi Zheng, Biwei Xie, Wei Li, Xiaoyu Wang, and Jianfeng Zhan. 2018 BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing (Bench18).

HPC AI500: A Benchmark Suite for HPC AI Systems. [PDF]
Zihan Jiang, Wanling Gao, Lei Wang, Xingwang Xiong, Yuchen Zhang, Xu Wen, Chunjie Luo, Hainan Ye, Xiaoyi Lu, Yunquan Zhang, Shengzhong Feng, Kenli Li, Weijia Xu, and Jianfeng Zhan. 2018 BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing (Bench18).

AIoT Bench: Towards Comprehensive Benchmarking Mobile and Embedded device Intelligence. [PDF]
Chunjie Luo, Fan Zhang, Cheng Huang, Xingwang Xiong, Jianan Chen, Lei Wang, Wanling Gao, Hainan Ye, Tong Wu, Runsong Zhou, and Jianfeng Zhan. 2018 BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing (Bench18).

Edge AIBench: Towards Comprehensive End-to-end Edge Computing Benchmarking. [PDF]
Tianshu Hao, Yunyou Huang, Xu Wen, Wanling Gao, Fan Zhang, Chen Zheng, Lei Wang, Hainan Ye, Kai Hwang, Zujie Ren, and Jianfeng Zhan. 2018 BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing (Bench18).

DCMix: Generating mixed workloads for the cloud data center. [PDF]
Xingwang Xiong, Lei Wang, Wanling Gao, Rui Ren, Ke Liu, Chen Zheng, Yu Wen, and Yi Liang. 2018 BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing (Bench18).

Data Motifs: A Lens Towards Fully Understanding Big Data and AI Workloads. [PDF]
Wanling Gao, Jianfeng Zhan, Lei Wang, Chunjie Luo, Daoyi Zheng, Fei Tang, Biwei Xie, Chen Zheng, Xu Wen, Xiwen He, Hainan Ye and Rui Ren. The 27th International Conference on Parallel Architectures and Compilation Techniques (PACT 2018).

BigDataBench: a Big Data Benchmark Suite from Internet Services. [PDF]
Lei Wang, Jianfeng Zhan, Chunjie Luo, Yuqing Zhu, Qiang Yang, Yongqiang He, WanlingGao, Zhen Jia, Yingjie Shi, Shujie Zhang, Cheng Zhen, Gang Lu, Kent Zhan, Xiaona Li, and Bizhu Qiu. The 20th IEEE International Symposium On High Performance Computer Architecture (HPCA-2014), February 15-19, 2014, Orlando, Florida, USA.

Data Motif-based Proxy Benchmarks for Big Data and AI Workloads. [PDF]
Wanling Gao, Jianfeng Zhan, Lei Wang, Chunjie Luo, Zhen Jia, Daoyi Zheng, Chen Zheng, Xiwen He, Hainan Ye, Haibin Wang, and Rui Ren. 2018 IEEE International Symposium on Workload Characterization (IISWC 2018).

BigDataBench: a Scalable and Unified Big Data and AI Benchmark Suite. [PDF]
Wanling Gao, Jianfeng Zhan, Lei Wang, Chunjie Luo, Daoyi Zheng, Rui Ren, Chen Zheng, Gang Lu, Jingwei Li, Zheng Cao, Shujie Zhang, and Haoning Tang. Technical Report, arXiv preprint arXiv:1802.08254, January 27, 2018.

BOPS, Not FLOPS! A New Metric and Roofline Performance Model For Datacenter Computing. [PDF]
Lei Wang, Jianfeng Zhan, Wanling Gao, Zihan Jiang, Rui Ren, Xiwen He, Chunjie Luo, Gang Lu, Jingwei Li. Technical Report, arXiv preprint arXiv:1801.09212, May 3, 2018.

Understanding Big Data Analytics Workloads on Modern Processors. [PDF]
Zhen Jia, Jianfeng Zhan, Lei Wang, Chunjie Luo, Wanling Gao, Yi Jin, Rui Han and Lixin Zhang. IEEE Transactions on Parallel and Distributed Systems, 28(6), 1797-1810, 2017.

Understanding Processors Design Decisions for Data Analytics in Homogeneous Data Centers. [PDF]
Zhen Jia, Wanling Gao, Yingjie Shi, Sally A. McKee, Jianfeng Zhan, Lei Wang, Lixin Zhang. IEEE Transactions on Big Data, 2017.

A Dwarf-based Scalable Big Data Benchmarking Methodology. [PDF]
Wanling Gao, Lei Wang, Jianfeng Zhan, Chunjie Luo, Daoyi Zheng, Zhen Jia, Biwei Xie, Chen Zheng, Qiang Yang, and Haibin Wang. arXiv preprint arXiv: 1711.03229

Characterizing data analysis workloads in data centers. [PDF]
Zhen Jia, Lei Wang, Jianfeng Zhan, Lixin Zhang, Chunjie Luo. 2013 IEEE International Symposium on Workload Characterization (IISWC 2013) (Best paper award).

Characterizing and Subsetting Big Data Workloads.[PDF]
Zhen Jia, Lei Wang, Jianfeng Zhan, Lixin Zhang, Chunjie Luo, Ninghui Sun. 2014 IEEE International Symposium on Workload Characterization (IISWC 2014)

Identifying Dwarfs Workloads in Big Data Analytics.  [PDF]
W Gao, C Luo, J Zhan, H Ye, X He, L Wang, Y Zhu, X Tian. 
arXiv preprint arXiv:1505.06872

BDGS: A Scalable Big Data Generator Suite in Big Data Benchmarking. [PDF]
Zijian Ming, Chunjie Luo, Wanling Gao, Rui Han, Qiang Yang, Lei Wang, and Jianfeng Zhan. In Advancing Big Data Benchmarks (pp. 138-154). Springer International Publishing.

Biographies

Jianfeng Zhan
Jianfeng Zhan is a Full Professor at Institute of Computing Technology, Chinese Academy of Sciences, and University of Chinese Academy of Sciences. He has supervised over 80 graduate students (both MS and Ph.D), post-docs, and engineers. His research interests cover a wide spectrum in the areas of high performance and distributed systems. He has made strong and effective efforts to transfer his academic research into advanced technology to impact general-purpose production systems. Currently, he is leading the research efforts for modern datacenter software stacks, including BigDataBench---an open source big data and AI benchmark suite, and RainForest--- an operating system for datacenter computing. Since the publication in HPCA 2014, BigDataBench is widely used in both academia and industry in the world. He has transferred more than 40 OS and distributed system patents to top companies. He founded BenchCouncil---a multidisciplinary international benchmark council and served as TPDS associate editor. More details about Prof. Zhan are available at http://www.benchcouncil.org/zjf.html

Wanling Gao
Wanling Gao is an Assistant Professor in computer science at the Institute of Computing Technology, Chinese Academy of Sciences and University of Chinese Academy of Sciences. Her research interests focus on big data benchmark and big data analytics. She received her B.S. degree in 2012 from Huazhong University of Science and Technology and her PhD degree in 2019 from Institute of Computing Technology, Chinese Academy of Sciences and University of Chinese Academy of Sciences in China.

Tianshu Hao
Tianshu Hao received the B.S. degree from Nankai University, Tianjin, China, in 2015. She is currently pursuing Ph. D. dergree in ICT, CAS. Her research interests focus on big data, edge computing, IoT and AI benchmarking.

Zihan Jiang
Zihan Jiang is a doctoral student in computer architecture at Institute of Computing Technology, Chinese Academy of Sciences and University of Chinese Academy of Sciences. His research interests includes AI benchmark and distributed deep learning. Currently, he works on HPC AI500 project.

Fei Tang
Fei Tang received the B.S. degree from Zhengzhou University, Zhengzhou, China, in 2016. He is currently pursuing Ph. D. dergree in ICT, CAS. His research interests focus on big data, benchmarking and search engine.