Skip navigation

BenchCouncil: International Open Benchmark Council


International Open Benchmarking Council (BenchCouncil) is a Non-profit Organization, which aims to promote multi-disciplinary benchmarking research and practice and foster collaboration and interaction between industry and academia.

What's New:

03/03/2020: AIBench Tutorial on ASPLOS 2020 Updated. This tutorial introduces the agile domain-specific benchmarking methodology, ten end-to-end application scenarios distilled from the industry-scale applications, AIBench framework, end-to-end, component, and micro benchmarks, and AIBench's value for software and hardware designer, micro-architectural researchers, and code developers. Additionally, the videos are provided to show how to use AIBench on a publicly available Testbed. All the AIBench slide presentations and hands-on tutorial videos are publicly available from Tutorial_Link. The separate link for each talk is also provided in Tutorial Website.

02/22/2020: AIBench Tutorial on HPCA 2020 Updated. This tutorial introduces the challenges, motivation, Industry Partners’ requirements, AIBench framework and benchmarks. Additionally, the videos are provided to show how to use AIBench on a publicly available Testbed. All the AIBench slide presentations and hands-on tutorial videos are publicly available from Tutorial_Link. The separate link for each talk is also provided in Tutorial Website.

12/20/2019: BenchCouncil AI Benchmark Specification Call for Comments: AIBench Specification, HPC AI500 Specification, Edge AIBench Specification, and AIoTBench Specification.

12/02/2019: 2020 BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing (Bench'20) is online (Nov 14-16, 2020 @ Atlanta, Georgia, USA).

12/02/2019: 2020 BenchCouncil International Federated Intelligent Computing and Block Chain Conferences (FICC, Jun 29-Jul 3, 2020 @ Qingdao, Shandong, China) is online. Embracing intelligent computing, FICC 2020 consists of four individual symposiums: Symposium on Intelligent Computers (IC 20), Symposium on Intelligent Medicine Technology (MedTech 20), Symposium on Block Chain and Intelligent Financial Technology (FinTech 20), and Symposium on Education Technology (EduTech 20).

12/02/2019: 2019 International Symposium on Chips (Chips 2019) registration open. Chips 2019 aims to discuss the advanced technologies and strategies about general chips, open-source chips, and intelligent chips, towards building an open chip ecosystem.

11/19/2019: BenchCouncil’s View On Benchmarking AI and Other Emerging Workloads (Technical Report, Slides presented by Prof. Jianfeng Zhan at BenchCouncil SC BoF). This paper outlines BenchCounci’s view on the challenges, rules, and vision of benchmarking modern workloads. We conclude the challenges of benchmarking modern workloads as FIDSS (Fragmented, Isolated, Dynamic, Servicebased, and Stochastic), and propose the PRDAERS benchmarking rules that the benchmarks should be specified in a paper-and-pencil manner, relevant, diverse, containing different levels of abstractions, specifying the evaluation metrics and methodology, repeatable, and scaleable. We believe proposing simple but elegant abstractions that help achieve both efficiency and general-purpose is the final target of benchmarking in future, which may be not pressing. In the light of this vision, we shortly discuss BenchCouncil’s related projects.

11/16/2019: Prof. Dr. Tony Hey honored with BenchCouncil Achievement Award. Prof. Dr. Tony Hey, the Chief Data Scientist at Rutherford Appleton Laboratory STFC, has been named the 2019 recipient of the International Open Benchmark Council (BenchCouncil) achievement award. Prof. Dr. Tony Hey is a fellow of ACM, the American Association for the Advancement of Science, and the Royal Academy of Engineering.

11/14/2019: Chips 2019 (Dec 18-20 @ Beijing, China) is online. Chips 2019 is organized by BenchCouncil and Bulletin of Chinese Academy of Sciences, aiming to discuss the key technology, software and hardware ecology of chips industry.

10/28/2019: Bench'19 Call for Participation. Bench'19 Program is updated.

10/23/2019: AIBench is open sourced.

10/23/2019: AIBench technical report updated.

10/16/2019: Bench 18 proceeding is online.

10/16/2019: Call for BenchCouncil Achievement Award Nomination.

10/06/2019: Call for HPC AI System Award Nomination.

09/26/2019: Performance numbers are updated.

09/17/2019: SC 19 BoF is online.

09/07/2019: Bench 19 Registration is open.

09/05/2019: Bench 19 Challenge Paper Submission is open.

09/02/2019: HPC AI500 and Edge AIBench are updated.

08/12/2019: AIBench, HPC AI500, AIoTBench, Edge AIBench are released.

08/12/2019: BigDataBench is updated.

06/27/2019 - 06/29/2019: 2019 BenchCouncil International Symposium on Intelligent Computers is held in Shenzhen.

05/01/2019: BenchCouncil Competition is online for registration.

03/13/2019: BenchCouncil 2019 is online.

03/07/2019: Bench'19 is online for paper submission.

02/17/2019: A full-day BigDataBench tutorial at HPCA'19, the slides are available now.

02/16/2019: Tutorial: BigDataBench 5.0 will be held in Washington D.C., US, co-located with hpca 19 conference.

12/20/2018: Tools for electricity load generator is available on website.

12/18/2018: Some slides of Benchmark Proposals present in Bench 18 have been available in website.

12/11/2018: Bench'18 Best Paper Winner: EC-Bench: Benchmarking Onload and Offload Erasure Coders on Modern Hardware Architectures Haiyang Shi, Xiaoyi Lu, Dhabaleswar K. (DK) Panda, The Ohio State University

12/01/2018: Bench19 (2019 International Symposium on Benchmarking, Measuring and Optimizing) will be held in Denver, US, prior to SC 19 conference.

11/15/2018: BenchCouncil 2019 Annual System Technical Conference (BenchCouncil ATC 2019). BenchCouncil ATC 2019 will be held in China in June 2019.

11/01/2018: BenchCouncil Transaction on Benchmarking, Measuring, and Optimizing (TBench) was founded. The first issue will be online in April 2019.

11/01/2018: BenchCouncil receives nine benchmarking proposals from famous research groups in China, US, and Europe.

06/01/2018: Bench 2018 is co-located with IEEE Big Data 2018