Publications

AIBench Publications

Project Homepage: https://www.benchcouncil.org/aibench.

* AIBench Training

AIBench Training: Balanced Industry-Standard AI Training Benchmarking.[PDF]

Fei Tang, Wanling Gao, Jianfeng Zhan, Chuanxin Lan, Xu Wen, Lei Wang, Chunjie Luo, Zheng Cao, Xiongwang Xiong, Zihan Jiang, Tianshu Hao, Fanda Fan, Fan Zhang, Yunyou Huang, Jianan Chen, Mengjia Du, Rui Ren, Chen Zheng, Daoyi Zheng, Haoning Tang, Kunlin Zhan, Biao Wang, Defei Kong, Minghe Yu, Chongkang Tan, Huan Li, Xinhui Tian, Yatao Li, Junchao Shao, Zhenyu Wang, Xiaoyu Wang, Jiahui Dai, Hainan Ye. 2021 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS 2021).

AIBench: Towards Scalable and Comprehensive Datacenter AI Benchmarking. [PDF]

Wanling Gao, Chunjie Luo, Lei Wang, Xingwang Xiong, Jianan Chen, Tianshu Hao, Zihan Jiang, Fanda Fan, Mengjia Du, Yunyou Huang, Fan Zhang, Xu Wen, Chen Zheng, Xiwen He,, Jiahui Dai, Hainan Ye, Zheng Cao, Zhen Jia, Kent Zhan, Haoning Tang, Daoyi Zheng, Biwei Xie, Wei Li, Xiaoyu Wang, and Jianfeng Zhan. 2018 BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing (Bench18).

* AIBench Scenario

AIBench Scenario: Scenario-distilling AI Benchmarking. [PDF]

Wanling Gao, Fei Tang, Jianfeng Zhan, Xu Wen, Lei Wang, Zheng Cao, Chuanxin Lan, Chunjie Luo, Xiaoli Liu, Zihan Jiang. The 30th International Conference on Parallel Architectures and Compilation Techniques (PACT 2021).

AIBench: An Industry Standard Internet Service AI Benchmark Suite. [PDF]

Wanling Gao, Fei Tang, Lei Wang, Jianfeng Zhan, Chunxin Lan, Chunjie Luo, Yunyou Huang, Chen Zheng, Jiahui Dai, Zheng Cao, Daoyi Zheng, Haoning Tang, Kunlin Zhan, Biao Wang, Defei Kong, Tong WW u, Minghe Yu, Chongkang Tan, Huan Li, Xinhui Tian, Yatao Li, Gang Lu, Junchao Shao, Zhenyu Wang, Xiaoyu Wang, and Hainan Ye. Technical Report, 2019.

* Edge AIBench

Edge AIBench: Towards Comprehensive End-to-end Edge Computing Benchmarking. [PDF]

Tianshu Hao, Yunyou Huang, Xu Wen, Wanling Gao, Fan Zhang, Chen Zheng, Lei Wang, Hainan Ye, Kai Hwang, Zujie Ren, and Jianfeng Zhan. 2018 BenchCouncil Internationall Symposium on Benchmarking, Measuring and Optimizing (Bench18)

* HPC AI500

HPC AI500 V2.0: The Methodology, Tools, and Metrics for Benchmarking HPC AI Systems. [PDF]

Zihan Jiang, Wanling Gao, Fei Tang, Lei Wang, Xingwang Xiong, Chunjie Luo, Chuanxin Lan, Hongxiao Li, Jianfeng Zhan. 2021 IEEE International Conference on Cluster Computing (CLUSTER 2021).

HPC AI500: Representative, Repeatable and Simple HPC AI Benchmarking. [PDF]

Zihan Jiang, Wanling Gao, Fei Tang, Xingwang Xiong, Lei Wang, Chuanxin Lan, Chunjie Luo, Hongxiao Li, Jianfeng Zhan. Revised Technical Report, 2021.

HPC AI500: A Benchmark Suite for HPC AI Systems. [PDF]

Zihan Jiang, Wanling Gao, Lei Wang, Xingwang Xiong, Yuchen Zhang, Xu Wen, Chunjie Luo, Hainan Ye, Xiaoyi Lu, Yunquan Zhang, Shengzhong Feng, Kenli Li, Weijia Xu, and Jianfeng Zhan. 2018 BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing (Bench18).

* AIoTBench

Comparison and Benchmarking of AI Models and Frameworks on Mobile Devices. [PDF]

Chunjie Luo, Xiwen He, Jianfeng Zhan, Kexin Zhao, Lei Wang, Wanling Gao, Shaopeng Dai. Technical Report, 2020.

AIoTBench: Towards Comprehensive Benchmarking Mobile and Embedded device Intelligence. [PDF]

Chunjie Luo, Fan Zhang, Cheng Huang, Xingwang Xiong, Jianan Chen, Lei Wang, Wanling Gao, Hainan Ye, Tong Wu, Runsong Zhou, and Jianfeng Zhan. 2018 BenchCouncil Intee rnational Symposium on Benchmarking, Measuring and Optimizing (Bench18)


Big Data Publications

BigDataBench Homepage: https://www.benchcouncil.org/BigDataBench.

BigDataBench: a Big Data Benchmark Suite from Internet Services. [PDF]

Lei Wang, Jianfeng Zhan, Chunjie Luo, Yuqing Zhu, Qiang Yang, Yongqiang He, WanlingGao, Zhen Jia, Yingjie Shi, Shujie Zhang, Cheng Zhen, Gang Lu, Kent Zhan, Xiaona Li, and Bizhu Qiu. The 20th IEEE International Symposium On High Performance Computer Architecture (HPCA-2014), February 15-19, 2014, Orlando, Florida, USA.

Understanding Big Data Analytics Workloads on Modern Processors. [PDF]

Zhen Jia, Jianfeng Zhan, Lei Wang, Chunjie Luo, Wanling Gao, Yi Jin, Rui Han and Lixin Zhang. IEEE Transactions on Parallel and Distributed Systems, 28(6), 1797-1810, 2017.

Understanding Processors Design Decisions for Data Analytics in Homogeneous Data Centers. [PDF]

Zhen Jia, Wanling Gao, Yingjie Shi, Sally A. McKee, Jianfeng Zhan, Lei Wang, Lixin Zhang. IEEE Transactions on Big Data, 2017.

Characterization and Architectural Implications of Big Data Workloads. [PDF]

Lei Wang, Jianfeng Zhan, Zhen Jia and Rui Han. arXiv:1506.07943 [cs.DC]

Identifying Dwarfs Workloads in Big Data Analytics. [PDF]

W Gao, C Luo, J Zhan, H Ye, X He, L Wang, Y Zhu, X Tian. arXiv preprint arXiv:1505.06872

BDGS: A Scalable Big Data Generator Suite in Big Data Benchmarking. [PDF]

Zijian Ming, Chunjie Luo, Wanling Gao, Rui Han, Qiang Yang, Lei Wang, and Jianfeng Zhan. In Advancing Big Data Benchmarks (pp. 138-154). Springer International Publishing.

Characterizing and Subsetting Big Data Workloads. [PDF]

Zhen Jia, Jianfeng Zhan, Wang Lei, Rui Han, Sally A. McKee, Qiang Yang, Chunjie Luo, and Jingwei Li. In 2014 IEEE International Symposium on Workload Characterization (IISWC). IEEE, 2014.

BigOP: generating comprehensive big data workloads as a benchmarking framework. [pdf]

Yuqing Zhu, Jianfeng Zhan, Chuliang Weng, Raghunath Nambiar, Jingchao Zhang, Xingzhen Chen, and Lei Wang. The 19th International Conference on Database Systems for Advanced Applications (DASFAA 2014), 2014.

BigDataBench: a Big Data Benchmark Suite from Web Search Engines.[pdf]

Wanling Gao, Yuqing Zhu, Zhen Jia, Chunjie Luo, Lei Wang, Jianfeng Zhan, Yongqiang He, Shiming Gong, Xiaona Li, Shujie Zhang, and Bizhu Qiu. Third Workshop on Architectures and Systems for Big Data (ASBD 2013) in conjunction with The 40th International Symposium on Computer Architecture, May 2013.

BigDataBench-MT: A Benchmark Tool for Generating Realistic Mixed Data Center Workloads. [PDF]

R Han, S Zhan, C Shao, J Wang, J Xu, LK John, L Wang, J Zhan. arXiv preprint arXiv:1504.02205

Characterizing data analysis workloads in data centers. [PDF]

Zhen Jia, Lei Wang, Jianfeng Zhan, Lixin Zhang, Chunjie Luo. 2013 IEEE International Symposium on Workload Characterization (IISWC 2013) (Best paper award).


Big Data and AI

Data Motifs: A Lens Towards Fully Understanding Big Data and AI Workloads. [PDF]

Wanling Gao, Jianfeng Zhan, Lei Wang, Chunjie Luo, Daoyi Zheng, Fei Tang, Biwei Xie, Chen Zheng, Xu Wen, Xiwen He, Hainan Ye, Rui Ren. The 27th International Conferencee on Parallel Architectures and Compilation Techniques (PACT18).

BigDataBench: a Scalable and Unified Big Data and AI Benchmark Suite. [PDF]

Wanling Gao, Jianfeng Zhan, Lei Wang, Chunjie Luo, Daoyi Zheng, Rui Ren, Chen Zheng, Gang Lu, Jingwei Li, Zheng Cao, Shujie Zhang, and Haoning Tang. Technical Reporr t, arXiv preprint arXiv:1802.08254, January 27, 2018.

Data Motif-based Proxy Benchmarks for Big Data and AI Workloads. [PDF]

Wanling Gao, Jianfeng Zhan, Lei Wang, Chunjie Luo, Zhen Jia, Daoyi Zheng, Chen Zheng, Xiwen He, Hainan Ye, Haibin Wang, and Rui Ren. IEEE International Symposium on Workload Characterization (IISWC 2018).

Characterization of real workloads of web search engines.[PDF]

Xi, H., Zhan, J., Jia, Z., Hong, X., Wang, L., Zhang, L., Lu, G. (2011, November). In 2011 IEEE International Symposium on Workload Characterization (IISWC), on (pp. 15-25). IEEE.


Cloud and Datacenters

DCMix: Generating mixed workloads for the cloud data center. [PDF]

Xingwang Xiong, Lei Wang, Wanling Gao, Rui Ren, Ke Liu, Chen Zheng, Yu Wen, and Yi Liang. 2018 BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing (Bench18).

BOPS, A New Computation-Centric Metric for Datacenter Computing. [PDF]

Lei Wang , Wanling Gao, Kaiyong Yang, and Zihan Jiang. Benchmarking, Measuring, and Optimizing. Bench 2019. Lecture Notes in Computer Science, vol 12093, Springer.

CloudRank-D: benchmarking and ranking cloud computing systems for data processing applications. [PDF]

Chunjie Luo, Jianfeng Zhan, Zhen Jia, Lei Wang, Gang Lu, Lixin Zhang, Cheng-Zhong Xu, Ninghui Sun. Front. Comput. Sci., 2012, 6(4): 347-362.


CPU

BENCHCPU Project Homepage: https://www.benchcouncil.org/benchcpu/index.html.

WPC

WPC Project Homepage: http://www.benchcouncil.org/WPC/.

Federated Learning

FLBench Project Homepage: https://www.benchcouncil.org/flbench.

FLBench: A Benchmark Suite for Federated Learning. [PDF]

Yuan Liang, Yange Guo, Yanxia Gong, Chunjie Luo, Jianfeng Zhan, and Yunyou Huang. Intelligent Computing and Block Chain: First BenchCouncil International Federated Conferences, FICC 2020, Qingdao, China, October 30–November 3, 2020, Revised Selected Papers. Springer Nature, 166.


Storage

OStoreBench Project Homepage: https://github.com/EVERYGO111/OStoreBench.

OStoreBench: Benchmarking Distributed Object Storage Systems Using Real-World Application Scenarios. [PDF]

Guoxin Kang, Defei Kong, Lei Wang, Jianfeng Zhan. Benchmarking, Measuring, and Optimizing. Bench 2020. Lecture Notes in Computer Science, vol 12614, Springer.


Energy Efficiency

Project Homepage: https://www.benchcouncil.org/electricity.

An electricity consumption model for synthesizing scalable electricity load curves. [PDF]

Y Huang, J Zhan, C Luo, L Wang, N Wang, D Zheng, F Fan, R Ren. Energy 169, 674-683.