BenchCouncil Instrument & Tools

To support the sustainable development of BenchCouncil, BenchCouncil provides a series of commercial tools and benchmarking services.

Big data benchmark tool

Big data benchmark tool – BigDataBench – supports measuring, benchmarking, and evaluating hardware systems, software systems, and business systems. It provides full life-cycle benchmarking services for the design, selection, acceptance review, expansion, and optimization of the big data systems. BigDataBench includes diverse workloads implemented with mainstream big data systems like Hadoop, Spark, Flink, and covered various types like search engine, e-commerce, social network, cognitive science, and medicine. From the perspective of dataset, BigDataBench uses real-world datasets and covers multiple data sources, e.g., text, table, graph, and image, and data types, e.g., structural, semi-structural, and un-structural. In addition, to support the scale-up and scale-out scalability, BigDataBench provides big data generator suite (BDGS) which supports user defined large-scale data size and preserves the characteristics of the real-world datasets.

  • Applications

    1、Provide guidance on the design and selection of the general-purpose big data software systems
    2、Provide guidance on the design and selection of the general-purpose big data hardware systems
    3、Provide guidance on the design and selection of the domain-specific big data software and hardware systems
    4、Provide guidance on the optimization of the big data algorithms and systems
    5、Support the research on big data algorithms and systems
    6、Provide big data benchmarking services and generate ranking list

 

AI benchmark tool: AIBench

BenchCouncil AI benchmark tool – AIBench – provides a customizable and extensible AI benchmarking framework, which adopts a hierarchical and modular design methodology and supports the expansion to customizable scenario and customizable workloads. According to different benchmarking requirements, the framework provides micro benchmarks that support Fine-grained analysis and component benchmarks that support the performance and quality evaluation of complex components. AIBench covers typical AI processing tasks, including the text, image, audio, and video processing.

  • Applications

    1、Provide guidance on the design and selection of the general-purpose AI software and hardware systems
    2、Provide guidance on the design and selection of the AI accelerators
    3、Provide guidance on the optimization of the AI systems
    4、Support the research on AI algorithms and systems
    5、Provide benchmarking services on AI training and generate ranking list
    6、Provide benchmarking services on AI inference and generate ranking list

 

Scenario simulator

Scenario simulator is a tool and platform that supports scenario-based simulation, evaluation, and verification. It provides the proxies of the large-scale complex scenarios like industrial internet and medical field and supports the evaluation and verification of the future scenarios. The features are as follows:
1、Supports the rapid simulation and emulation of complex application scenarios and even future scenarios, and reduces the complexity of the real-world scenarios by 2-3 orders of magnitude;
2、Supports the expansion to customizable scenarios according to the user requirements;
3、Provides the simulation and evaluation of industrial internet and medical scenarios;
4、Provides the simulation of the complex scenarios across the IoTs, edges, and datacenters;
5、Provides one-click deployment and installation, saving the familiar cost of different frameworks and systems, and providing installation templates;
6、Supports multi-user and multi-task simulation;
7、Full life-cycle management, covering the design, selection, acceptance review, and system online.
8、Automatic performance analysis, providing the guidance on the deployment and optimization.

The real-world scenarios usually contain various AI and non-AI modules, and consequently forms long and complicated execution paths. Previous tools focus on a few kernels, or components, or submodules, fails to model an industry-scale application scenario’s overall critical paths, and further obtains biased suggestions and conclusions. Scenario simulator distills and abstracts real-world application scenarios, and can reflect the overall performance of the real-world scenarios.

  • Applications

    1、Support the demonstration and display of research/ exploration project achievements
    2、Support integrated demonstration and display of multiple project achievements
    3、Provide testbed for evaluating advanced technologies and supporting research exploration and experimentation
    4、Support the evaluation and verification of the early-stage and middle-stage complex systems
    5、Support the evaluation and verification of the future systems or projects
    6、Support the simulation and verification of the extremely complex systems

 

Whole-picture workload characterization (WPC) tool

Traditional workload characterization methods focus on single-level analysis like the microarchitectural level on a specific micro architecture using performance numbers, binary stream level based on a specific ISA, and intermediate representation (IR level). These methods lack the whole-picture analysis across the whole stack, and thus fail to capture the workloads’ inherent characteristics. BenchCouncil WPC tool integrates microarchitecture-dependent, microarchitecture-independent, and ISA-independent characterization methodologies. It performs a whole-picture analysis on hierarchical profile data across Intermediate Representation (IR), ISA, and microarchitecture to sum up the inherent workload characteristics and understand the reasons behind the numbers.

  • Applications

    1、Workload characterization on processor design
    2、Workload characterization on ISA design
    3、Provide processor benchmarking services and generate ranking list
    4、Support the research on computer architecture
    5、Analyze the hotspot functions and find the performance bottlenecks
    6、Software and hardware co-design

 

Network analyzer

Large-scale network simulation and traffic analysis are of great significance not only for the complex applications’ network analysis and optimization, but also for the architecture design and optimization of the switch chips.

BenchCouncil network analyzer supports the simulation of the large-scale datacenter network topologies and the analysis of network traffics. The features are as follows:
1、Supports the simulation of diverse network topologies;
2、Provides the network traffic models, including Internet services, AI, big data, etc;
3、Provide the network traffic replay tool that preserves the characteristics of the real-world applications and supports microsecond even nanoseconds precision.
4、Provides the benchmarking services on switch chips and covers comprehensive performance metrics including switching, congestion control, delay jitter, and burst absorption performance.

  • Applications

    1、Support the evaluation and verification of large-scale network systems
    2、Support the researches on network systems
    3、Support the development and implementation of network systems
    4、Provide guidance on the selection, evaluation, and optimization of network systems
    5、Support the display of network system achievements
    6、Support the simulation and verification of future network systems

 

Contact

If you want to know more, please contact us

Location:

Zhongguancun, Haidian District, Beijing, PRC