AIBench Training: Balanced AI Benchmarking, BenchCouncil

 

RANKING

We evaluate CPUs, GPUs and other AI accelerators using AIBench Training RPR subset. BenchCouncil will publish the performance numbers periodically, more intelligent chips and accelerators will be evaluated. BenchCouncil welcomes everyone join the benchmarking and submit their results who is interested in the performance of AI systems and architectures.

Metrics

BenchCouncil reports performance number, performance & quality number (time-to-quality), and energy consumption number (energy-to-quality) using AIBench Training RPR subset.

  • Time-to-quality Number of AIBench Training RPR Subset.

    As the training time to a state-of-the-art quality requires a lot of execution time, for performance & quality ranking, we only choose AIBench Training RPR subset for reducing the cost just like that the HPC Top500 ranking only reports three benchmarks.

  • Energy-to-quality Number of AIBench Training RPR Subset.

    As the training time to a state-of-the-art accuracy requires a lot of execution time, for energy ranking, we only choose AIBench Training RPR Subset for reducing the cost just like that the HPC Top500 ranking only reports three benchmarks.

  • Throughput Number of full benchmarks of AIBench.

    We evaluate CPUs, GPUs and other AI accelerators using the AIBench inference benchmark. We run the benchmarks using optimized parameter settings to achieve the quality of referenced paper and report the throughput performance.

Time-to-quality Numbers


Energy-to-quality number is available soon.

Inference Performance on Single GPU

To evaluate the inference performance of intelligent chips, we also evaluate the inference time on single GPU card. We find that the V100 also has the best inference performance, which performs 4 times better than the others at most. The inference performance of Titan XP is colse to V100 and has higher price–performance ratio. Likewise, the inference of partial AI workloads also have high memory requirements, such as object detection.