BenchCouncil Instrument & Tools

Big data benchmark tool

Big data benchmark tool – BigDataBench – supports measuring, benchmarking, and evaluating hardware systems, software systems, and business systems. It provides full life-cycle benchmarking services for the design, selection, acceptance review, expansion, and optimization of the big data systems. BigDataBench includes diverse workloads implemented with mainstream big data systems like Hadoop, Spark, Flink, and covered various types like search engine, e-commerce, social network, cognitive science, and medicine. From the perspective of dataset, BigDataBench uses real-world datasets and covers multiple data sources, e.g., text, table, graph, and image, and data types, e.g., structural, semi-structural, and un-structural. In addition, to support the scale-up and scale-out scalability, BigDataBench provides big data generator suite (BDGS) which supports user defined large-scale data size and preserves the characteristics of the real-world datasets.

AI benchmark tool: AIBench

BenchCouncil AI benchmark tool – AIBench – provides a customizable and extensible AI benchmarking framework, which adopts a hierarchical and modular design methodology and supports the expansion to customizable scenario and customizable workloads. According to different benchmarking requirements, the framework provides micro benchmarks that support Fine-grained analysis and component benchmarks that support the performance and quality evaluation of complex components. AIBench covers typical AI processing tasks, including the text, image, audio, and video processing.

Scenario simulator

Scenario simulator is a tool and platform that supports scenario-based simulation, evaluation, and verification. It provides the proxies of the large-scale complex scenarios like industrial internet and medical field and supports the evaluation and verification of the future scenarios. The features are as follows:

1、Supports the rapid simulation and emulation of complex application scenarios and even future scenarios, and reduces the complexity of the real-world scenarios by 2-3 orders of magnitude;
2、Supports the expansion to customizable scenarios according to the user requirements;
3、Provides the simulation and evaluation of industrial internet and medical scenarios;
4、Provides the simulation of the complex scenarios across the IoTs, edges, and datacenters;
5、Provides one-click deployment and installation, saving the familiar cost of different frameworks and systems, and providing installation templates;
6、Supports multi-user and multi-task simulation;
7、Full life-cycle management, covering the design, selection, acceptance review, and system online.
8、Automatic performance analysis, providing the guidance on the deployment and optimization.

The real-world scenarios usually contain various AI and non-AI modules, and consequently forms long and complicated execution paths. Previous tools focus on a few kernels, or components, or submodules, fails to model an industry-scale application scenario’s overall critical paths, and further obtains biased suggestions and conclusions. Scenario simulator distills and abstracts real-world application scenarios, and can reflect the overall performance of the real-world scenarios.

Whole-picture workload characterization (WPC) tool

Traditional workload characterization methods focus on single-level analysis like the microarchitectural level on a specific micro architecture using performance numbers, binary stream level based on a specific ISA, and intermediate representation (IR level). These methods lack the whole-picture analysis across the whole stack, and thus fail to capture the workloads’ inherent characteristics. BenchCouncil WPC tool integrates microarchitecture-dependent, microarchitecture-independent, and ISA-independent characterization methodologies. It performs a whole-picture analysis on hierarchical profile data across Intermediate Representation (IR), ISA, and microarchitecture to sum up the inherent workload characteristics and understand the reasons behind the numbers.

Network analyzer

Large-scale network simulation and traffic analysis are of great significance not only for the complex applications’ network analysis and optimization, but also for the architecture design and optimization of the switch chips.

BenchCouncil network analyzer supports the simulation of the large-scale datacenter network topologies and the analysis of network traffics. The features are as follows:
1、Supports the simulation of diverse network topologies;
2、Provides the network traffic models, including Internet services, AI, big data, etc;
3、Provide the network traffic replay tool that preserves the characteristics of the real-world applications and supports microsecond even nanoseconds precision.
4、Provides the benchmarking services on switch chips and covers comprehensive performance metrics including switching, congestion control, delay jitter, and burst absorption performance.