BenchCouncil: International Open Benchmark Council

A BenchCouncil View on Benchmarking Emerging and Future Computing (PDF, published by TBench). This paper presents a unifying benchmark definition, a conceptual framework, and a traceable and supervised learning-based benchmarking methodology for benchmark science and engineering.

BenchCouncil’s View On Benchmarking AI and Other Emerging Workloads (Technical Report, Slides presented by Prof. Jianfeng Zhan at BenchCouncil SC BoF). This paper outlines BenchCounci’s view on the challenges, rules, and vision of benchmarking modern workloads.

BenchCouncil Top Level Projects

(1) BenchCouncil BigDataBench

Homepage: http://www.benchcouncil.org/BigDataBench/

.

(2) BenchCouncil AIBench

Homepage: http://www.benchcouncil.org/aibench/.

• AIBench Training - nineteen representative AI tasks with state-of-the-art models to guarantee diversity and representativeness.
  • HPC AI500 - large-scale HPC AI system benchmarking.
• AIBench Inference - nineteen representative AI tasks with state-of-the-art models to guarantee diversity and representativeness.
  • AIoTBench - an open-source benchmark suite for Mobile and Embedded device Intelligence.
• AIBench Micro - intensively-used hotspot functions for simulation-based architecture researches.
• AIBench Synthetic - complementary to real-world benchmarks, with scalable problem sizes to model learning dynamics.

Specification:

• AIBench specification
• AIBench ranking specification
• Edge AIBench specification
• HPC AI500 specification
• HPC AI500 ranking specification
• AIoTBench specification

(3) BenchCouncil ScenarioBench

Homepage: http://www.benchcouncil.org/scenariobench/

• AIBench Scenario - a scenario benchmark suite modeling AI-augmented Internet service scenarios.
• Edge AIBench - a scenario benchmark suite modeling end-to-end performance across IoT, edge, and Datacenter.

(4) BenchCouncil Clinical AIBench

Homepage: https://www.benchcouncil.org/clinical_aibench/

(5) BenchCouncil SAIBench

Homepage: https://www.computercouncil.org/SAIBench/

BenchCouncil Incubator Projects

(1) BenchCouncil BENCHCPU

Homepage: http://www.benchcouncil.org/benchcpu/

.

(2) BenchCouncil WPC

Homepage: http://www.benchcouncil.org/WPC/

.

(3) BenchCouncil PowerSystemBench

Homepage: https://www.benchcouncil.org/electricity.html

.

(4) BenchCouncil OLxPBench

Homepage: https://www.benchcouncil.org/olxpbench

.