News: Bench18 Call for Papers (Co-located with Big Data 2018). BigDataBench 4.0 released on April 1. Technical Report on BigDataBench 4.0, Big Data and AI Motif, and BOPS: new metric for Datacenter Computing.

Summary

As architecture, system, data management, and machine learning communities pay greater attention to innovative big data and data-driven artificial intelligence (in short, AI) algorithms, architecture, and systems, the pressure of benchmarking rises. However, complexity, diversity, frequently changed workloads, and rapid evolution of big data, especially AI systems raise great challenges in benchmarking. First, for the sake of conciseness, benchmarking scalability, portability cost, reproducibility, and better interpretation of performance data, we need understand what are the most time-consuming classes of unit of computation among big data and AI workloads. Second, for the sake of fairness, the benchmarks must include diversity of data and workloads. Third, for co-design of software and hardware, the benchmarks should be consistent across different communities.

We consider each big data and AI workload as a pipeline of one or more classes of units of computation performed on different initial or intermediate data inputs, each of which we call a data motif. For the first time, among a wide variety of big data and AI workloads, we identify eight data motifs— including Matrix, Sampling, Logic, Transform, Set, Graph, Sort and Statistic computation, each of which captures the common requirements of each class of unit of computation while being reasonably divorced from individual implementations. Significantly different from the traditional kernels, a data motif’s behaviors are affected by the sizes, patterns, types, and sources of different data inputs; Moreover, it reflects not only computation patterns, memory access patterns, but also disk and network I/O patterns.

As a multi-discipline research and engineering effort, i.e., architecture, system, data management, and machine learning communities from both industry and academia, we set up an open-source big data and AI benchmark suite—BigDataBench. The current version BigDataBench 4.0 provides 13 representative real-world data sets and 47 benchmarks. Other than creating a new benchmark or proxy for every possible workload, we propose using data motif-based benchmarks—the combination of eight data motifs—to represent diversity of big data and AI workloads. Our benchmark suite includes micro benchmarks, each of which is a single data motif, components benchmarks, which consist of the data motif combinations, and end-to-end application benchmarks, which are the combinations of component benchmarks.

The benchmarks cover seven workload types including AI, online services, offline analytics, graph analytics, data warehouse, NoSQL, and streaming from important application domains, i. e., search engines, social networks, e-commerce, multimedia processing and bioinformatics. Meanwhile, data sets have great impacts on workloads behaviors and running performance (CGO’18). Hence, data varieties are considered with the whole spectrum of data types including structured, semi-structured, and unstructured data. Currently, the included data sources are text, graph, table, and image data. Using real data sets as the seed, the data generators—BDGS— generate synthetic data by scaling the seed data while keeping the data characteristics of raw data.

To achieve the consistency of benchmarks across different communities, we absorb state-of-the-art algorithms from the machine learning communities that considers the model’s prediction accuracy. For the benchmarking requirements of system and data management communities, we provide diverse implementations using the state-of-the-art techniques. For offline analytics, we provide Hadoop, Spark, Flink and MPI implementations. For graph analytics, we provide Hadoop, Spark GraphX, Flink Gelly and GraphLab implementations. For AI, we provide TensorFlow and Caffe implementations. For data warehouse, we provide Hive, Spark-SQL and Impala implementations. For NoSQL, we provide MongoDB and HBase implementations. For streaming, we provide Spark streaming and JStorm implementations.

For the architecture community, whatever early in the architecture design process or later in the system evaluation, it is time-consuming to run a comprehensive benchmark suite. The complex software stacks of the big data and AI workloads aggravate this issue. To tackle this challenge, we propose the data motif-based simulation benchmarks for architecture communities, which speed up runtime 100 times while preserving system and micro-architectural characteristic accuracy. Also, we propose another methodology to reduce the benchmarking cost, we select a small number of representative benchmarks, called the BigDataBench subset according to workload characteristics from an architecture perspective. We provide the BigDataBench architecture subset on the MARSSx86, gem5, and Simics simulator versions, respectively.

On a typical state-of-practice processor: Intel Xeon E5- 2620 V3, we also perform comprehensive characterizations on the benchmarks of seven workload types in BigDataBench 4.0 in addition to traditional benchmarks like SPECCPU, PARSEC and HPCC, in a hierarchical manner and drill down on five levels, using the Top-Down analysis from an architecture perspective. We have the following observations. First, as listed in Figure 1, the ILP (instruction-level parallelism) of the AI benchmarks 2 is 1.26 on average, slightly lower than SPECCPU (1.32). The MLP (memory-level parallelism) of AI is 2.65, similar with HPCC (2.78). Big data has lower ILP (0.85 on average) and MLP (1.86 on average) than AI for almost all types, except that Hive based data warehouse has slightly higher ILP than AI. Further, their performance vary across workload types and software stacks.

Figure 1 Average Execution Performance.

Second, as listed in Figure 2, in terms of uppermost-level breakdown, AI reflect similar pipeline behaviors with the traditional benchmarks, with approximately equal retiring (35% v.s. 39.8%), bad speculation (6.3% v.s. 6.1%), frontend bound (both about 9%), and backend bound percentages (49.7% v.s. 45.1%). The frontend bound of big data is more severe than that of traditional benchmarks (9% on average). However, we notice that the frontend bound varies across different workload types. NoSQL has the highest percentage of 35%, while data warehouse has 25% and the other has only 15% on average. Please see our technical report for more details.

Figure 2 Uppermost Level Breakdown of All Benchmarks.

 

 

To model and reproduce multi-application or multi-user scenarios on Cloud or datacenters, we provide the multi-tenancy version of BigDataBench, which allows flexible setting and replaying of mixed workloads according to the real workload traces—the Facebook, Google and Sogou traces.

Together with several industry partners, including Telecom Research Institute Technology, Huawei, Intel (China), Microsoft (China), IBM CDL, Baidu, Sina, INSPUR, ZTE and etc, we also release China’s first industry standard big data benchmark suite—-BigDataBench-DCA, which is a subset of BigDataBench.

Why BigDataBench?

As shown in Table 1, among seven desired properties, we can find that BigDataBench is more sophisticated than the other state-of-art big data benchmarks.

Table 1: The Differences of BigDataBench from Other Benchmarks Suites.

	Benchmarking Target	Methodology	Application domains	Workload types	Workloads	Real data sets and scalable data sets	Software stacks
BigDataBench	Big data and AI systems and architecture	Data Motif-based	five	Seven [1]	forty seven	13 real data sets; 6 scalable data sets	sixteen
BigBench 2.0	Big data systems	Application model	one	five	Proposal	Proposal	Proposal
CloudSuite 3.0	Cloud services	Popularity	N/A	four	eight	3 data generators	three
HiBench 6.0	Big data systems	Popularity	N/A	six	nineteen	Random generate or with specific distribution	five
CALDA	MapReduce system and parallel DBMSs	Popularity	N/A	one	five	N/A	three
YCSB	Cloud serving systems	Performance model	N/A	one	six	N/A	four
LinkBench	Database system	Application model	N/A	one	ten	one data generator	two
AMP Benchmarks	Data analytic systems	Popularity	N/A	one	four	N/A	five
Fathom	AI systems	Popularity	N/A	one	eight	N/A	one

^[1]The seven workload types are online service, offline analytics, graph analytics, artificial intelligence, data warehouse, NoSQL, and streaming.

What’s New?

BigDataBench 4.0 proposes a data motif-based benchmarking methodology, and provides a set of micro, component and end-to-end application benchmarks, to fulfill different benchmarking requirements. Currently, BigDataBench includes 13 real-world data sets, and 47 big data benchmarks, covering seven workload types, including AI, online service, offline analytics, graph analytics, data warehouse, NoSQL, and streaming. Also, for simulation-based architecture research, we also provide data motif-based simulation benchmarks that preserve high system and micro-architectural characteristic accuracy (above 90% on average), and largely shorten (100 times speedup) the simulation time.

Methodology

Figure 7 summarizes our data motif-based benchmarking methodology for BigDataBench 4.0, separating the specification from implementation. Circling around the data motifs, we define the specification of micro, component, and end-to-end application benchmarks.

Figure 7 BigDataBench Benchmarking Methodology.

Micro Benchmark Specification. Data motifs are fundamental concepts and units of computation among a majority of big data and AI workloads. We design a suite of micro benchmarks, each of which is a single data motif, as listed in Table 2. We also notice that these micro benchmark implementations have different characteristics, i.e., CPU-intensive, memory-intensive or I/O-intensive.

Component Benchmark Specification. Data motif combinations can compose original complex workloads. In addition to micro benchmarks consisting of a single data motif, we also consider component benchmarks, which are representative workloads in different application domains. Component benchmarks are combinations of one or more data motifs using a DAG-like structure, as listed in Table 3. For example, SIFT is a combination of five data motifs, including matrix, sampling, transform, sort and statistic computations.

Application Benchmark Specification. To model an application domain, we define end-to-end application benchmark specification considering user characteristics and processing logic, based on the real process of an application domain. Due to the complexity and difficulty to benchmark a real application domain, we simplify and model the primary process of an application domain, and provide portable and usable end-to-end benchmarks. We use the combination of component benchmarks to represent the processing logic. For example, for online service, we generate queries considering query number, rate, distribution and locality to reflect the user characteristics.

Benchmarks

BigDataBench is in fast expansion and evolution. Currently, we proposed benchmarks specifications modeling five typical application domains. The current version BigDataBench 4.0 includes 13 real-world data sets and 47 big data workloads, covering seven types. Table 2 summarizes the real-world data sets and scalable data generation tools included into BigDataBench 4.0, covering the whole spectrum of data types, including structured, semi-structured, and unstructured data, and different data sources, including text, graph, image, audio, video and table data. Table 3 and Table 4 present the micro benchmarks and component benchmarks in BigDataBench 4.0 from perspectives of involved data motif, application domain, workload type, data set and software stack. For some end users, they may just pay attention to big data application of a specific type. For example, they want to perform an apples-to- apples comparison of software stacks for offline analytics. They only need to choose benchmarks with the type of offline analytics. But if the users want to measure or compare big data systems and architecture, we suggest they cover all benchmarks.

Table 2.The summary of data sets and data generation tools

	DATA SETS	DATA SIZE	SCALABLE DATA SET
1	Wikipedia Entries	4,300,000 English articles（unstructured text)	Text Generator of BDGS
2	Amazon Movie Reviews	7,911,684 reviews(semi-structured text)	Text Generator of BDGS
3	Google Web Graph	875713 nodes, 5105039 edges(unstructured graph)	Graph Generator of BDGS
4	Facebook Social Network	4039 nodes, 88234 edges (unstructured graph)	Graph Generator of BDGS
5	E-commerce Transaction Data	table1:4 columns,38658 rows. table2: 6columns, 242735 rows(structured table)	Table Generator of BDGS
6	ProfSearch Person Resumes	278956 resumes(semi-structured table)	Table Generator of BDGS
7	CIFAR-10	60000 color images with the dimension of 32*32	Ongoing development
8	ImageNet	ILSVRC2014 DET image dataset(unstructured image)	Ongoing development
9	LSUN	One million labelled images, classified into 10 scene categories and 20 object categories	Ongoing development
10	TED Talks	Translated TED talks provided by IWSLT evaluation campaign	Ongoing development
11	SoGou Data	the corpus and search query data from So-Gou Labs(unstructured text)	Ongoing development
12	MNIST	handwritten digits database which has 60,000 training examples and 10,000 test examples(unstructured image)	Ongoing development
13	MovieLens Dataset	User’s score data for movies, which has 9,518,231 training examples and 386,835 test examples(semi-structured text)	Ongoing development

Table 3. The summary of the micro benchmarks in BigDataBench 4.0

Micro Benchmark	Involved Data Motif	Application Domain	Workload Type	Date Set	Software Stack
Sort	Sort	SE, SN, EC, MP, BI [1]	Offline analytics	Wikipedia entries	Hadoop, Spark, Flink, MPI
Grep	Set	SE, SN, EC, MP, BI	Offline analytics	Wikipedia entires	Hadoop, Spark, Flink, MPI
			Streaming	Random generate	Spark streaming
WordCount	Basic statistics	SE, SN, EC, MP, BI	Offline analytics	Wikipedia entires	Hadoop, Spark, Flink, MPI
MD5	Logic	SE, SN, EC, MP, BI	Offline analytics	Wikipedia entires	Hadoop, Spark, MPI
Connected Component	Graph	SN	Graph analytics	Facebook social network	Hadoop, Spark, Flink, GraphLab, MPI
RandSample	Sampling	SE, MP, BI	Offline analytics	Wikipedia entires	Hadoop, Spark, MPI
FFT	Transform	MP	Offline analytics	Two-dimensional matrix	Hadoop, Spark, MPI
Matrix Multiply	Matrix	SE, SN, EC, MP, BI	Offline analytics	Two-dimensional matrix	Hadoop, Spark, MPI
Read	Set	SE, SN, EC	NoSQL	ProfSearch resumes	HBase, MongoDB
Write	Set	SE, SN, EC	NoSQL	ProfSearch resumes	HBase, MongoDB
Scan	Set	SE, SN, EC	NoSQL	ProfSearch resumes	HBase, MongoDB
Convolution	Transform	SN, EC, MP, BI	AI	Cifar, ImageNet	TensorFlow, Caffe
Fully Connected	Matrix	SN, EC, MP, BI	AI	Cifar, ImageNet	TensorFlow, Caffe
Relu	Logic	SN, EC, MP, BI	AI	Cifar, ImageNet	TensorFlow, Caffe
Sigmoid	Matrix	SN, EC, MP, BI	AI	Cifar, ImageNet	TensorFlow, Caffe
Tanh	Matrix	SN, EC, MP, BI	AI	Cifar, ImageNet	TensorFlow, Caffe
MaxPooling	Sampling	SN, EC, MP, BI	AI	Cifar, ImageNet	TensorFlow, Caffe
AvgPooling	Sampling	SN, EC, MP, BI	AI	Cifar, ImageNet	TensorFlow, Caffe
CosineNorm	Basic Statistics	SN, EC, MP, BI	AI	Cifar, ImageNet	TensorFlow, Caffe
BatchNorm	Basic Statistics	SN, EC, MP, BI	AI	Cifar, ImageNet	TensorFlow, Caffe
Dropout	Sampling	SN, EC, MP, BI	AI	Cifar, ImageNet	TensorFlow, Caffe

^[1]SE (Search Engine), SN (Social Network), EC (e-commerce), BI (Bioinformatics), MP (Multimedia Processing).
Table 4. The summary of the component benchmarks in BigDataBench 4.0

Component Benchmark	Involved Data Motif	Application Domain	Workload Type	Date Set	Software Stack
Xapian Server	Get, Put, Post	SE	Online service	Wikipedia entries	Xapian
PageRank	Matrix, Sort, Basic statistics, Graph	SE	Graph analytics	Google web graph	Hadoop, Spark, Flink, GraphLab, MPI
Index	Logic, Sort, Basic statistics, Set	SE	Offline analytics	Wikipedia entries	Hadoop, Spark
Rolling top words	Sort, Basic statistics	SN	Streaming	Random generate	Spark streaming, JStorm
Kmeans	Matrix, Sort, Basic statistics	SE, SN, EC, MP, BI	Offline analytics	Facebook social network	Hadoop, Spark, Flink, MPI
			Streaming	Random generate	Spark streaming
Collaborative Filtering	Graph, Matrix	EC	Offline analytics	Amazon movie review	Hadoop, Spark
			Streaming	MovieLens dataset	JStorm
Naive Bayes	Basic statistics, Sort	SE, SN, EC	Offline analytics	Amazon movie review	Hadoop, Spark, Flink, MPI
SIFT	Matrix, Sampling, Transform, Sort	MP	Offline analytics	ImageNet	Hadoop, Spark, MPI
LDA	Matrix, Graph, Sampling	SE	Offline analytics	Wikipedia entries	Hadoop, Spark, MPI
OrderBy	Set, Sort	EC	Data warehouse	E-commerce transaction	Hive, Spark-SQL, Impala
Aggregation	Set, Basic statistics	EC	Data warehouse	E-commerce transaction	Hive, Spark-SQL, Impala
Project	Set	EC	Data warehouse	E-commerce transaction	Hive, Spark-SQL, Impala
Filter	Set	EC	Data warehouse	E-commerce transaction	Hive, Spark-SQL, Impala
Select	Set	EC	Data warehouse	E-commerce transaction	Hive, Spark-SQL, Impala
Union	Set	EC	Data warehouse	E-commerce transaction	Hive, Spark-SQL, Impala
Alexnet	Matrix, Transform, Sampling, Logic, Basic statistics	SN, MP, BI	AI	Cifar, ImageNet	TensorFlow, Caffe
Googlenet		SN, MP, BI	AI	Cifar, ImageNet	TensorFlow, Caffe
Resnet		SN, MP, BI	AI	Cifar, ImageNet	TensorFlow, Caffe
Inception Resnet V2		SN, MP, BI	AI	Cifar, ImageNet	TensorFlow, Caffe
VGG16		SN, MP, BI	AI	Cifar, ImageNet	TensorFlow, Caffe
DCGAN		SN, MP, BI	AI	LSUN	TensorFlow, Caffe
WGAN		SN, MP, BI	AI	LSUN	TensorFlow, Caffe
GAN	Matrix, Sampling, Logic, Basic Statistics	SN, MP, BI	AI	LSUN	TensorFlow, Caffe
Seq2Seq		SN, EC, BI	AI	TED Talks	TensorFlow, Caffe
Word2vec	Matrix, Basic statistics, Logic	SE, SN, EC	AI	Wikipedia entries, Sogou data	TensorFlow, Caffe

Evolution

As shown in Figure 2, the evolution of BigDataBench has gone through three major stages: At the first stage, we released three benchmarks suites, BigDataBench 1.0 (6 workloads from Search engine), DCBench 1.0 (11 workloads from data analytics), and CloudRank 1.0(mixed data analytics workloads).

At the second stage, we merged the previous three benchmark suites and release BigDataBench 2.0 , through investigating the top three important application domains from internet services in terms of the number of page views and daily visitors. BigDataBench 2.0 includes 6 real-world data sets, and 19 big data workloads with different implementations, covering six application scenarios: micro benchmarks, Cloud OLTP, relational query, search engine, social networks, and e-commerce. Moreover, BigDataBench 2.0 provides several big data generation tools–BDGS– to generate scalable big data, e.g, PB scale, from small-scale real-world data while preserving their original characteristics.

BigDataBench 3.0 is a multidisciplinary effort. It includes 6 real-world, 2 synthetic data sets, and 32 big data workloads, covering micro and application benchmarks from typical application domains, e. g., search engine, social networks, and e-commerce. As to generating representative and variety of big data workloads, BigDataBench 3.0 focuses on units of computation that frequently appear in Cloud OLTP, OLAP, interactive and offline analytics.

Figure 2: BigDataBench Evolution

Previous releases

BigDataBench 3.2 http://prof.ict.ac.cn/BigDataBench/old/3.2/

BigDataBench 3.1 http://prof.ict.ac.cn/BigDataBench/old/3.1/

BigDataBench 3.0 http://prof.ict.ac.cn/BigDataBench/old/3.0/

BigDataBench 2.0 http://prof.ict.ac.cn/BigDataBench/old/2.0/

BigDataBench 1.0 http://prof.ict.ac.cn/BigDataBench/old/1.0/

DCBench 1.0 http://prof.ict.ac.cn/DCBench/

CloudRank 1.0 http://prof.ict.ac.cn/CloudRank/

Handbook

Handbook of BigDataBench [BigDataBench-handbook]

Q &A

More questions & answers are available from the handbook of BigDataBench.

Contacts (Email)

• gaowanling@ict.ac.cn
• lijingwei@mail.bafst.com
• zhanjianfeng@ict.ac.cn
• wl@ncic.ac.cn

People

• Prof. Jianfeng Zhan, ICT, CAS
• Dr. Lei Wang, ICT, CAS
• Wanling Gao, ICT, CAS
• Chunjie Luo, ICT, CAS
• Qiang Yang, BAFST
• Jingwei Li, BAFST
• Xinhui Tian, ICT, CAS
• Dr. Gang Lu, BAFST
• Xinlong Lin, BAFST
• Rui Ren, ICT, CAS
• Dr, Rui Han, ICT, CAS
• Daoyi Zheng, ICT, CAS
• Dr. Chen Zheng, ICT, CAS
• Zheng Cao, Alibaba
• Shujie Zhang, Huawei
• Haoning Tang, Tencent

• Alumni

• Dr. Zhen Jia, Princeton University
• Hainan Ye, BAFST
• Dr. Yingjie Shi,
• Zijian Ming, Tencent
• Yuanqing Guo, Sohu
• Yongqiang He, Dropbox
• Kent Zhan, WUBA
• Xiaona Li

• Bizhu Qiu, Yahoo

License

BigDataBench is available for researchers interested in big data. Software components of BigDataBench are all available as open-source software and governed by their own licensing terms. Researchers intending to use BigDataBench are required to fully understand and abide by the licensing terms of the various components. BigDataBench is open-source under the Apache License, Version 2.0. Please use all files in compliance with the License. Our BigDataBench Software components are all available as open-source software and governed by their own licensing terms. If you want to use our BigDataBench you must understand and comply with their licenses. Software developed externally (not by BigDataBench group)

• Boost: http://www.boost.org/doc/libs/1_43_0/more/getting_started/unix-variants.html
• GCC: http://gcc.gnu.org/releases.html
• GSL: http://www.gnu.org/software/gsl/
• Graph500 : http://www.graph500.org/referencecode
• Hadoop: http://www.apache.org/licenses/LICENSE-2.0
• HBase : http://hbase.apache.org/
• Hive: http://hive.apache.org/
• Impala: https://github.com/cloudera/impala
• MySQL : http://www.mysql.com/
• Mahout: http://www.apache.org/licenses/LICENSE-2.0
• Mpich: http://www.mpich.org/
• Nutch : http://www.apache.org/licenses/LICENSE-2.0
• Parallel Boost Graph Library : http://www.osl.iu.edu/research/pbgl/software/
• Spark: http://spark.incubator.apache.org/
• Shark: http://shark.cs.berkeley.edu/
• Scala: http://www.scala-lang.org/download/2.9.3.html
• Zookeeper: http://zookeeper.apache.org/

Software developed internally (by BigDataBench group) BigDataBench_4.0 License BigDataBench_4.0 Suite Copyright (c) 2013-2018, ICT Chinese Academy of Sciences All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

• Redistribution of source code must comply with the license and notice disclaimers
• Redistribution in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimers in the documentation and/or other materials provided by the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE ICT CHINESE ACADEMY OF SCIENCES BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.