|
ScenarioBench: Modeling, Characterizing, and Optimizing Ultra-scale Real-world or Future Applications and Systems Using Benchmarks |
|
|
ScenarioBench OverviewThe goal of ScenarioBench is to propose methodology, tools, and metrics to model, characterize, and optimize ultra-scale real-world or future applications and systems using the benchmarks. Instead of using real-world applications or implementing a full-fledged application from scratch, we propose the permutations of essential tasks as a scenario benchmark. The goal is to identify the critical paths and primary modules of a real-world scenario since they consume the most system resources and are the core focuses for system design and optimization. Each scenario benchmark distillates the crucial attributes of an industry-scale application and reduces the side effect of the latter’s complexity in terms of huge code size, extreme deployment scale, and complex execution paths. MotivationModern real-world application scenarios like Internet services are complex, which raises severe challenges on benchmarking, evaluating, and optimization. Their code sizes and deployment scales are about three to four orders of magnitude larger than traditional workloads. Traditional benchmarking methodologies using microbenchmarks or component benchmarks only capture a small fraction of execution paths and several separate components of real-world application scenarios, and cannot reflect the critical paths and primary modules, thus may lead to error-prone conclusions. While profiling has been applied to various aspects of benchmarking, we are not aware of evidence demonstrating it can holistically evaluate a large and complex system. For example, a few publicly available performance models or insights observed through profiling do not work across different systems or architectures for Internet services. As there are no publicly available benchmarks, the state-of-the-art and state-of-the-practice are advanced only by the research staff within various service providers, which poses a considerable obstacle for our communities towards developing an open and mature research field. TargetsScenarioBench targets to provide proxies to industry-scale real-world applications scenarios or future application scenarios. Each scenario benchmark models the critical paths of a real-world application scenario as a permutation of the multiple essential modules.
MethodologyWe formalize a real-world or future application scenario as a Directed Acyclic Graph-based model (in short, DAG model) and propose the rules to distill it into a permutation of essential tasks, e.g., AI and non-AI tasks, as a scenario benchmark. We call it a scenario-distilling methodology. Our methodology is to identify the critical path and primary modules of a real-world scenario since they consume the most system resources and are the core focuses for system design and optimization. We will extend to white-box (apply to industry users who can access the code) and black-box (without input) methodologies in the future. In cooperation with industry partners, we extract several vital scenario benchmarks. We design and implement a reusing benchmark framework. Based on the framework, we implement the scenario benchmarks–E-commerce Search Intelligence and Online Translation Intelligence. Each scenario benchmark reduces the complexity by one or two orders of magnitude to its counterpart real-world application scenario. It is easier to achieve the latter’s efficient implementation to avoid misleading evaluations using a suboptimal implementation. For a real-world application scenario, the complex software evolution also hinders high-performance implementation. ArtifactsAt present, ScenarioBench includes two scenario benchmark suites---AIBench Scenario, which focuses on complex Internet services application scenarios augmented with AI techniques, and Edge AIBench, which focuses on complex application scenarios across IoT, edge, and Datacenter. AIBench Scenario AIBench Scenario is a benchmarks suite modeling AI-augmented Internet service scenarios, including E-commerce Search Intelligence and Online Translation Intelligence. Each scenario benchmark models the critical paths of a real-world application scenario as a permutation of the AI and non-AI modules. Artifacts: Download from Zenodo, Download from GitHub. Edge AIBench Edge AIBench is a scenario benchmark suite modeling end-to-end performance across IoT, edge, and Datacenter. Four representative edge scenarios are covered, including ICU Patient Monitor, Surveillance Camera, Smart Home, and Autonomous Vehicle. Artifacts: Edge AIBench 1.0, Edge AIBench 2.0. Contributors
Prof. Jianfeng Zhan, BenchCouncil & ICT, Chinese Academy of Sciences Publications
* AIBench ScenarioAIBench Scenario: Scenario-distilling AI Benchmarking. [PDF] Wanling Gao, Fei Tang, Jianfeng Zhan, Xu Wen, Lei Wang, Zheng Cao, Chuanxin Lan, Chunjie Luo, Xiaoli Liu, Zihan Jiang. The 30th International Conference on Parallel Architectures and Compilation Techniques (PACT 2021). AIBench: An Industry Standard Internet Service AI Benchmark Suite. [PDF] Wanling Gao, Fei Tang, Lei Wang, Jianfeng Zhan, Chunxin Lan, Chunjie Luo, Yunyou Huang, Chen Zheng, Jiahui Dai, Zheng Cao, Daoyi Zheng, Haoning Tang, Kunlin Zhan, Biao Wang, Defei Kong, Tong Wu, Minghe Yu, Chongkang Tan, Huan Li, Xinhui Tian, Yatao Li, Gang Lu, Junchao Shao, Zhenyu Wang, Xiaoyu Wang, and Hainan Ye. Technical Report, 2019. * Edge AIBenchAI-oriented Medical Workload Allocation for Hierarchical Cloud/Edge/Device Computing. [PDF] Tianshu Hao, Jianfeng Zhan, Kai Hwang, Wanling Gao, Xu Wen. The 21st IEEE/ACM international Symposium on Cluster, Cloud and. Internet Computing (CCGrid 2021). Edge AIBench: Towards Comprehensive End-to-end Edge Computing Benchmarking. [PDF] Tianshu Hao, Yunyou Huang, Xu Wen, Wanling Gao, Fan Zhang, Chen Zheng, Lei Wang, Hainan Ye, Kai Hwang, Zujie Ren, and Jianfeng Zhan. 2018 BenchCouncil International Symposium on Benchmarking, Measuring and Optimizing (Bench18). UsersAlibaba |