As diverse communities pay great attention to innovative AI or machine learning algorithms, architecture, and systems, the pressure of benchmarking rises. However, complexity, diversity, frequently changed workloads, and rapid evolution of AI workloads and systems raise great challenges in AI benchmarking. The aim of this BoF is to discuss how to build a comprehensive AI benchmark suite across different communities with an emphasis on data and workload distributions among HPC, data center, Edge, and IoT.

(1) HPC AI500: A Benchmark Suite for HPC AI Systems. [Homepage], [Paper]

(2) AIBench: An Industry Standard Internet Service AI Benchmark Suite. [Homepage], [Technical Report], [Bench18]

(3) AIoTBench: Towards Comprehensive Benchmarking Mobile and Embedded device Intelligence. [Homepage], [Paper]

(4) Edge AIBench: Towards Comprehensive End-to-end Edge Computing Benchmarking. [Homepage], [Paper]

(5) FakeBench: a benchmark for detecting fake images and videos.

The source code of BenchCouncil AI Benchmarks are hosted on BenchCouncil code management system---BenchHub. The hand-on tutorials use BenchCouncil Testbed as platform, which is the world's first large-scale and open artificial intelligence testbed.

We will give a BoF forum on BenchCouncil AI Benchmarks at SC19 in Denver, Colorado, USA.

November 19, 2019 (Tuesday), 05:15 pm - 06:45 pm (An hour and a half)

ROOM: 503-504

As a multi-discipline, i.e., system, architecture, data management and machine learning, research and engineering effort from both industry and academia, BenchCouncil AI Benchmarks is a comprehensive AI benchmark suite for HPC, Datacenter, Edge, and IoT.

HPC AI500 is a benchmark suite for evaluating HPC systems that running scientific DL workloads. Each workload from HPC AI500 bases on real scientific DL applications and covers the most representative scientific fields, namely climate analysis, cosmology, high energy physics, gravitational wave physics and computational biology. Currently, we choose 18 scientific DL benchmarks (For details, see Specification) from application scenarios, datasets, and software stack. Furthermore, we propose a set of metrics of comprehensively evaluating the HPC systems, considering both accuracy, performance as well as power and cost. In addition, we provide a scalable reference implementation of HPC AI500.

AIBench is the first industry scale AI benchmark suite, joint with seventeen industry partners. First, we present a highly extensible, configurable, and flexible benchmark framework, containing multiple loosely coupled modules like data input, prominent AI problem domains, online inference, offline training and automatic deployment tool modules. We analyze typical AI application scenarios from three most important Internet services domains, and then we abstract and identify sixteen prominent AI problem domains, including classification, image generation, text-to-text translation, image-to-text, image-to- image, speech-to-text, face embedding, 3D face recognition, object detection, video prediction, image compression, recommendation, 3D object reconstruction, text summarization, spatial transformer, and learning to rank. We implement sixteen component benchmarks for those AI problem domains, and further profile and implement twelve fundamental units of computation across different component benchmarks as the micro benchmarks.

Edge AIBench is a benchmark suite for end-to-end edge computing including four typical application scenarios: ICU Patient Monitor, Surveillance Camera, Smart Home, and Autonomous Vehicle, which consider the complexity of all edge computing AI scenarios. In addition, Edge AIBench provides an end-to-end application benchmarking framework, including train, validate and inference stages. Table 1 shows the component benchmarks of Edge AIBench. Edge AIBench provides an end-to-end application benchmarking, consisting of train, inference, data collection and other parts using a general three-layer edge computing framework.

AIoTBench is a comprehensive benchmark suite to evaluate the AI ability of mobile and embedded devices. Our benchmark 1) covers different application domains, e.g. image recognition, speech recognition and natural language processing; 2) covers different platforms, including Android devices and Raspberry Pi; 3) covers different development tools, including TensorFlow and Caffe2; 4) offers both end-to-end application workloads and micro workloads.