BenchCouncil’s View On Benchmarking AI and Other Emerging Workloads (Technical Report, Slides presented by Prof. Jianfeng Zhan at BenchCouncil SC BoF). This paper outlines BenchCounci’s view on the challenges, rules, and vision of benchmarking modern workloads.
BenchCouncil Top Level Projects
(1) BenchCouncil BigDataBench
Homepage: http://www.benchcouncil.org/BigDataBench/
.(2) BenchCouncil AIBench
Homepage: http://www.benchcouncil.org/aibench/.
-
AIBench Scenario - proxies to industry-scale real-world applications scenarios.
- Edge AIBench - an instance of the scenario benchmark suites, modeling end-to-end performance across IoT, edge, and Datacenter.
-
AIBench Training - nineteen representative AI tasks with state-of-the-art models to guarantee diversity and representativeness.
- HPC AI500 - large-scale HPC AI system benchmarking.
-
AIBench Inference - nineteen representative AI tasks with state-of-the-art models to guarantee diversity and representativeness.
- AIoTBench - an open-source benchmark suite for Mobile and Embedded device Intelligence.
- AIBench Micro - intensively-used hotspot functions for simulation-based architecture researches.
- AIBench Synthetic - complementary to real-world benchmarks, with scalable problem sizes to model learning dynamics.
Specification:
- AIBench specification
- AIBench ranking specification
- Edge AIBench specification
- HPC AI500 specification
- HPC AI500 ranking specification
- AIoTBench specification
BenchCouncil Incubator Projects
(1) BenchCouncil BENCHCPU
Homepage: http://www.benchcouncil.org/benchcpu/
.(2) BenchCouncil WPC
Homepage: http://www.benchcouncil.org/WPC/
.(3) BenchCouncil FLBench
Homepage: http://www.benchcouncil.org/flbench.html/
.(4) BenchCouncil EleGeneration
.