Skip navigation

AIBench: A Comprehensive AI Benchmark Suite for Datacenter, HPC, Edge, and AIoT

 

Publications

BenchCouncil’s View On Benchmarking AI and Other Emerging Workloads (Technical Report, Slides presented by Prof. Jianfeng Zhan at BenchCouncil SC BoF). This paper outlines BenchCounci’s view on the challenges, rules, and vision of benchmarking modern workloads.


AIBench Publications

Project Homepage: https://www.benchcouncil.org/aibench.

* AIBench Training

AIBench Training: Balanced Industry-Standard AI Training Benchmarking.[PDF]

Fei Tang, Wanling Gao, Jianfeng Zhan, Chuanxin Lan, Xu Wen, Lei Wang, Chunjie Luo, Zheng Cao, Xiongwang Xiong, Zihan Jiang, Tianshu Hao, Fanda Fan, Fan Zhang, Yunyou Huang, Jianan Chen, Mengjia Du, Rui Ren, Chen Zheng, Daoyi Zheng, Haoning Tang, Kunlin Zhan, Biao Wang, Defei Kong, Minghe Yu, Chongkang Tan, Huan Li, Xinhui Tian, Yatao Li, Junchao Shao, Zhenyu Wang, Xiaoyu Wang, Jiahui Dai, Hainan Ye. 2021 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS 2021).

AIBench: Towards Scalable and Comprehensive Datacenter AI Benchmarking. [PDF]

Wanling Gao, Chunjie Luo, Lei Wang, Xingwang Xiong, Jianan Chen, Tianshu Hao, Zihan Jiang, Fanda Fan, Mengjia Du, Yunyou Huang, Fan Zhang, Xu Wen, Chen Zheng, Xiwen He,, Jiahui Dai, Hainan Ye, Zheng Cao, Zhen Jia, Kent Zhan, Haoning Tang, Daoyi Zheng, Biwei Xie, Wei Li, Xiaoyu Wang, and Jianfeng Zhan. 2018 BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing (Bench18).


* AIBench Scenario

AIBench Scenario: Scenario-distilling AI Benchmarking. [PDF]

Wanling Gao, Fei Tang, Jianfeng Zhan, Xu Wen, Lei Wang, Zheng Cao, Chuanxin Lan, Chunjie Luo, Xiaoli Liu, Zihan Jiang. The 30th International Conference on Parallel Architectures and Compilation Techniques (PACT 2021).

AIBench: An Industry Standard Internet Service AI Benchmark Suite. [PDF]

Wanling Gao, Fei Tang, Lei Wang, Jianfeng Zhan, Chunxin Lan, Chunjie Luo, Yunyou Huang, Chen Zheng, Jiahui Dai, Zheng Cao, Daoyi Zheng, Haoning Tang, Kunlin Zhan, Biao Wang, Defei Kong, Tong WW u, Minghe Yu, Chongkang Tan, Huan Li, Xinhui Tian, Yatao Li, Gang Lu, Junchao Shao, Zhenyu Wang, Xiaoyu Wang, and Hainan Ye. Technical Report, 2019.


* Edge AIBench

Edge AIBench: Towards Comprehensive End-to-end Edge Computing Benchmarking. [PDF]

Tianshu Hao, Yunyou Huang, Xu Wen, Wanling Gao, Fan Zhang, Chen Zheng, Lei Wang, Hainan Ye, Kai Hwang, Zujie Ren, and Jianfeng Zhan. 2018 BenchCouncil Internationall Symposium on Benchmarking, Measuring and Optimizing (Bench18)


* HPC AI500

HPC AI500 V2.0: The Methodology, Tools, and Metrics for Benchmarking HPC AI Systems. [PDF]

Zihan Jiang, Wanling Gao, Fei Tang, Lei Wang, Xingwang Xiong, Chunjie Luo, Chuanxin Lan, Hongxiao Li, Jianfeng Zhan. 2021 IEEE International Conference on Cluster Computing (CLUSTER 2021).

HPC AI500: Representative, Repeatable and Simple HPC AI Benchmarking. [PDF]

Zihan Jiang, Wanling Gao, Fei Tang, Xingwang Xiong, Lei Wang, Chuanxin Lan, Chunjie Luo, Hongxiao Li, Jianfeng Zhan. Revised Technical Report, 2021.

HPC AI500: The Methodology, Tools, Roofline Performance Models, and Metrics for Benchmarking HPC AI Systems. [PDF]

Zihan Jiang, Lei Wang, Xingwang Xiong, Wanling Gao, Chunjie Luo, Fei Tang, Chuanxin Lan, Hongxiao Li, Jianfeng Zhan. Technical Report, 2020.

HPC AI500: A Benchmark Suite for HPC AI Systems. [PDF]

Zihan Jiang, Wanling Gao, Lei Wang, Xingwang Xiong, Yuchen Zhang, Xu Wen, Chunjie Luo, Hainan Ye, Xiaoyi Lu, Yunquan Zhang, Shengzhong Feng, Kenli Li, Weijia Xu, and Jianfeng Zhan. 2018 BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing (Bench18).


* AIoTBench

Comparison and Benchmarking of AI Models and Frameworks on Mobile Devices. [PDF]

Chunjie Luo, Xiwen He, Jianfeng Zhan, Kexin Zhao, Lei Wang, Wanling Gao, Shaopeng Dai. Technical Report, 2020.

AIoTBench: Towards Comprehensive Benchmarking Mobile and Embedded device Intelligence. [PDF]

Chunjie Luo, Fan Zhang, Cheng Huang, Xingwang Xiong, Jianan Chen, Lei Wang, Wanling Gao, Hainan Ye, Tong Wu, Runsong Zhou, and Jianfeng Zhan. 2018 BenchCouncil Intee rnational Symposium on Benchmarking, Measuring and Optimizing (Bench18)