A BenchCouncil View on Benchmarking Emerging and Future Computing (PDF, published by TBench).
This paper presents a unifying benchmark definition, a conceptual framework, and a traceable and supervised learning-based benchmarking methodology for benchmark science and engineering.
BenchCouncil’s View On Benchmarking AI and Other Emerging Workloads (Technical Report, Slides presented by Prof. Jianfeng Zhan at BenchCouncil SC BoF). This paper outlines BenchCounci’s view on the challenges, rules, and vision of benchmarking modern workloads.
BenchCouncil Top Level Projects
(1) BenchCouncil BigDataBench
Homepage: http://www.benchcouncil.org/BigDataBench/
.(2) BenchCouncil AIBench
Homepage: http://www.benchcouncil.org/aibench/.
-
AIBench Training - nineteen representative AI tasks with state-of-the-art models to guarantee diversity and representativeness.
- HPC AI500 - large-scale HPC AI system benchmarking.
-
AIBench Inference - nineteen representative AI tasks with state-of-the-art models to guarantee diversity and representativeness.
- AIoTBench - an open-source benchmark suite for Mobile and Embedded device Intelligence.
- AIBench Micro - intensively-used hotspot functions for simulation-based architecture researches.
- AIBench Synthetic - complementary to real-world benchmarks, with scalable problem sizes to model learning dynamics.
Specification:
- AIBench specification
- AIBench ranking specification
- Edge AIBench specification
- HPC AI500 specification
- HPC AI500 ranking specification
- AIoTBench specification
(3) BenchCouncil ScenarioBench
Homepage: http://www.benchcouncil.org/scenariobench/
- AIBench Scenario - a scenario benchmark suite modeling AI-augmented Internet service scenarios.
- Edge AIBench - a scenario benchmark suite modeling end-to-end performance across IoT, edge, and Datacenter.
(4) BenchCouncil Clinical AIBench
Homepage: https://www.benchcouncil.org/clinical_aibench/
(5) BenchCouncil SAIBench
Homepage: https://www.computercouncil.org/SAIBench/
BenchCouncil Incubator Projects
(1) BenchCouncil BENCHCPU
Homepage: http://www.benchcouncil.org/benchcpu/
.(2) BenchCouncil WPC
Homepage: http://www.benchcouncil.org/WPC/
.(3) BenchCouncil PowerSystemBench
Homepage: https://www.benchcouncil.org/electricity.html
.(4) BenchCouncil OLxPBench
Homepage: https://www.benchcouncil.org/olxpbench
.