Skip navigation

ScenarioBench: Modeling, Characterizing, and Optimizing Ultra-scale Real-world or Future Applications and Systems Using Benchmarks

 

ScenarioBench Overview

The goal of ScenarioBench is to propose methodology, tools, and metrics to model, characterize, and optimize ultra-scale real-world or future applications and systems using the benchmarks.

Instead of using real-world applications or implementing a full-fledged application from scratch, we propose the permutations of essential tasks as a scenario benchmark. The goal is to identify the critical paths and primary modules of a real-world scenario since they consume the most system resources and are the core focuses for system design and optimization. Each scenario benchmark distillates the crucial attributes of an industry-scale application and reduces the side effect of the latter’s complexity in terms of huge code size, extreme deployment scale, and complex execution paths.

Motivation

Modern real-world application scenarios like Internet services are complex, which raises severe challenges on benchmarking, evaluating, and optimization. Their code sizes and deployment scales are about three to four orders of magnitude larger than traditional workloads.

Traditional benchmarking methodologies using microbenchmarks or component benchmarks only capture a small fraction of execution paths and several separate components of real-world application scenarios, and cannot reflect the critical paths and primary modules, thus may lead to error-prone conclusions.

While profiling has been applied to various aspects of benchmarking, we are not aware of evidence demonstrating it can holistically evaluate a large and complex system. For example, a few publicly available performance models or insights observed through profiling do not work across different systems or architectures for Internet services. As there are no publicly available benchmarks, the state-of-the-art and state-of-the-practice are advanced only by the research staff within various service providers, which poses a considerable obstacle for our communities towards developing an open and mature research field.

Targets

ScenarioBench targets to provide proxies to industry-scale real-world applications scenarios or future application scenarios. Each scenario benchmark models the critical paths of a real-world application scenario as a permutation of the multiple essential modules.

  • Provide tools and platforms to support the simulation, evaluation, and verification of complex real-world applications scenarios and systems.
  • Quickly build and simulate large-scale complex application scenarios and enable rapid deployment of innovative technologies.
  • Simulate cooperative interaction and evolution of multiple complex scenarios, e.g., data center, edge, IoT.
  • Explore the simulation, evaluation, and verification of future application scenarios and systems.

Methodology

We formalize a real-world or future application scenario as a Directed Acyclic Graph-based model (in short, DAG model) and propose the rules to distill it into a permutation of essential tasks, e.g., AI and non-AI tasks, as a scenario benchmark. We call it a scenario-distilling methodology. Our methodology is to identify the critical path and primary modules of a real-world scenario since they consume the most system resources and are the core focuses for system design and optimization. We will extend to white-box (apply to industry users who can access the code) and black-box (without input) methodologies in the future. In cooperation with industry partners, we extract several vital scenario benchmarks. We design and implement a reusing benchmark framework. Based on the framework, we implement the scenario benchmarks–E-commerce Search Intelligence and Online Translation Intelligence. Each scenario benchmark reduces the complexity by one or two orders of magnitude to its counterpart real-world application scenario. It is easier to achieve the latter’s efficient implementation to avoid misleading evaluations using a suboptimal implementation. For a real-world application scenario, the complex software evolution also hinders high-performance implementation.

Artifacts

At present, ScenarioBench includes two scenario benchmark suites---AIBench Scenario, which focuses on complex Internet services application scenarios augmented with AI techniques, and Edge AIBench, which focuses on complex application scenarios across IoT, edge, and Datacenter.

AIBench Scenario

AIBench Scenario is a benchmarks suite modeling AI-augmented Internet service scenarios, including E-commerce Search Intelligence and Online Translation Intelligence. Each scenario benchmark models the critical paths of a real-world application scenario as a permutation of the AI and non-AI modules.

Artifacts: Download from Zenodo, Download from GitHub.

Edge AIBench

Edge AIBench is a scenario benchmark suite modeling end-to-end performance across IoT, edge, and Datacenter. Four representative edge scenarios are covered, including ICU Patient Monitor, Surveillance Camera, Smart Home, and Autonomous Vehicle.

Artifacts: Download from GitHub.

Contributors

Prof. Jianfeng Zhan, BenchCouncil & ICT, Chinese Academy of Sciences
Wanling Gao, BenchCouncil & ICT, Chinese Academy of Sciences
Fei Tang, ICT, Chinese Academy of Sciences
Tianshu Hao, ICT, Chinese Academy of Sciences
Xu Wen, ICT, Chinese Academy of Sciences
Lei Wang, BenchCouncil & ICT, Chinese Academy of Sciences
Zheng Cao, Alibaba
Chuanxin Lan, ICT, Chinese Academy of Sciences
Chunjie Luo, BenchCouncil & ICT, Chinese Academy of Sciences
Yunyou Huang, Guangxi Normal University
Fan Zhang, ICT, Chinese Academy of Sciences
Chen Zheng, Institute of Software, Chinese Academy of Sciences
Hainan Ye, BenchCouncil
Kai Hwang, Chinese University of Hong Kong (Shenzhen)
Xiaoli Liu, Alibaba
Zihan Jiang, ICT, Chinese Academy of Sciences
Zujie Ren, Zhejiang Lab

Publications

* AIBench Scenario

AIBench Scenario: Scenario-distilling AI Benchmarking. [PDF]

Wanling Gao, Fei Tang, Jianfeng Zhan, Xu Wen, Lei Wang, Zheng Cao, Chuanxin Lan, Chunjie Luo, Xiaoli Liu, Zihan Jiang. The 30th International Conference on Parallel Architectures and Compilation Techniques (PACT 2021).

AIBench: An Industry Standard Internet Service AI Benchmark Suite. [PDF]

Wanling Gao, Fei Tang, Lei Wang, Jianfeng Zhan, Chunxin Lan, Chunjie Luo, Yunyou Huang, Chen Zheng, Jiahui Dai, Zheng Cao, Daoyi Zheng, Haoning Tang, Kunlin Zhan, Biao Wang, Defei Kong, Tong Wu, Minghe Yu, Chongkang Tan, Huan Li, Xinhui Tian, Yatao Li, Gang Lu, Junchao Shao, Zhenyu Wang, Xiaoyu Wang, and Hainan Ye. Technical Report, 2019.

* Edge AIBench

AI-oriented Medical Workload Allocation for Hierarchical Cloud/Edge/Device Computing. [PDF]

Tianshu Hao, Jianfeng Zhan, Kai Hwang, Wanling Gao, Xu Wen. The 21st IEEE/ACM international Symposium on Cluster, Cloud and. Internet Computing (CCGrid 2021).

Edge AIBench: Towards Comprehensive End-to-end Edge Computing Benchmarking. [PDF]

Tianshu Hao, Yunyou Huang, Xu Wen, Wanling Gao, Fan Zhang, Chen Zheng, Lei Wang, Hainan Ye, Kai Hwang, Zujie Ren, and Jianfeng Zhan. 2018 BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing (Bench18).

Users

Alibaba