Summary

AIBench Synthetic is a part of the AIBench benchmarking suite. AIBench Synthetic and other AIBench benchmarks together resolve the AI benchmarking challenges. AI benchmarks like AIBench Training use real-world workloads as benchmarks to evaluate AI systems. They can reflect the actual performance in the real world. However, due to the limited network architectures, hyperparameters, etc., real-world workloads only can measure limited workload characterizations.

Meanwhile, state-of-the-art AI algorithms evolve very fast, which causes the real-world workloads' short shelf-life while adding or updating real-work workloads are time-costly. AIBench Synthetic uses representative building blocks like a fully connected layer and residual block combined with different hyperparameters to generate many networks, covering computation-intensive and memory-intensive workloads. These generated networks have more workload characterizations than real-word workloads and can represent future AI models, which resolve the real-word workloads' short shelf-life problem.

The Methodology

We extract the most widely used building blocks from nineteen representative workloads from AIBench Training. These building blocks include FC, CNN, RNN, Attention, etc. Based on these building blocks, we can construct various workloads by search different neural architectures in a search space. This search space is determined by many hyperparameters like layer numbers, node numbers, stride and padding sizes. Meanwhile, we train the generated models to convergence.

Contributors

Prof. Jianfeng Zhan, ICT, Chinese Academy of Sciences, and BenchCouncil    
Dr. Wanling Gao, ICT, Chinese Academy of Sciences    
Fei Tang, ICT, Chinese Academy of Sciences    
Dr. Lei Wang, ICT, Chinese Academy of Sciences    
Chunjie Luo, ICT, Chinese Academy of Sciences    
Chuanxin Lan, ICT, Chinese Academy of Sciences

License

AIBench is available for researchers interested in AI. Software components of AIBench are all available as open-source software and governed by their own licensing terms. Researchers intending to use AIBench are required to fully understand and abide by the licensing terms of the various components. AIBench is open-source under the Apache License, Version 2.0. Please use all files in compliance with the License. Our AIBench Software components are all available as open-source software and governed by their own licensing terms. If you want to use our AIBench you must understand and comply with their licenses. Software developed externally (not by AIBench group)

  • Redistribution of source code must comply with the license and notice disclaimers
  • Redistribution in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimers in the documentation and/or other materials provided by the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE ICT CHINESE ACADEMY OF SCIENCES BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.