Homepage
Benchmarking, measuring, and optimizing are fundamental activities in human being's lives. Starting as nine BPOE Workshop series and SDBA Workshop in conjunction with ASPLOS, VLDB, and ICS, this symposium (Bench 18) is organized by International Open Benchmarking Council (BenchCouncil) and dedicated to benchmarking, measuring, and optimizing complex systems, including (but not limited to) Big Data, AI, chainblock, datacenter, cloud and warehouse-scale computing, high performance computing, Mobile Robotics, edge and fog computing，Big Scientific data, IoT, and the other miscellaneous things, i.e., education systems, financial systems, and power systems. In addition, we encourage publishing benchmark specifications and the summaries of the open-source benchmarking projects.
Program
Preliminary Bench'18 program is relased. (PDF Version)
Slides:
Benchmarking Opportunities and Challenges: present and future of BenchCouncil (Prof. Jianfeng Zhan)
Big Data Benchmarking: Applications and Systems (Prof. Geoffrey Fox)
DataMotif: A Benchmark Proposal for Big Data and AI (Dr. Wanling Gao)
Characterizing and Benchmarking Deep Learning Systems on Modern Data Center Architectures (Prof. Xiaoyi Lu)
A Benchmark proposal for Datacenter Computing (Dr. Chen Zheng)
MLPerf: The Vision Behind an ML Benchmark Suite for Measuring the Performance of ML Software Frameworks, ML Hardware Accelerators, and ML Cloud and Edge Platforms (Prof. Vijay Janapa Reddi)
TS-benchmark: a benchmark proposal for time series databases (Prof. Yueguo Chen)
Benchmarking SpMV on Many-Core Architecture (Dr. Zhen Jia)
DCMIX: Generating Mixed Workloads for the Cloud Data Center (Prof. Yi Liang)
PeakBench: A Benchmark for Scalable Transaction Processing Systems (Prof. Weining Qian)
EC-Bench: Benchmarking Onload and Offload Erasure Coders on Modern Hardware Architectures (Haiyang Shi)
IBDB: A Benchmark Suite for Industrial Big Data System (Yijian Cheng)
Power Characterization of Memory Intensive Applications: Analysis and Implications
Scalability Evaluation of Big Data Processing Services in Clouds
Testing Raft-replicated Database Systems
Important Dates
Submission Deadline (Submission Site)
| Time | Event |
|---|---|
| Electronic submission of full papers | |
| Nov 9, 2018 | Notification of paper acceptance |
| Dec 10-12, 2018 | Conference |
| Feb 9, 2019 | Camera-ready of extended papers |
Call for Benchmarking Proposals
Benchmarking proposals are solicited for BenchCouncil open meeting, which will take place in Seattle, WA, on Dec 10-13. Benchmark proposals are solicited on any topic that is relevant to Bench 2018 audience, including (but not limited to):
Artificial Intelligence
NewSQL and distributed database systems
Blockchain
Datacenter, cloud and warehouse-scale computing
High performance computing
Mobile Robotics
Egde and fog computing
Big Scientific Data
Internet of Things (IoT)
Education, financial and power system
Title.
Authors and contact info.
1 paragraph abstract suitable for benchmark publicity.
biography per author suitable for publicity.
Once a proposal is accepted, at least one of the authors should present their proposal. Meanwhile, a breakout session will be organized to discuss the benchmarking targets, major challenges, and plans. Finally, for each invited talk, a breakout report will be presented by the same speaker.
After the conference, the authors should form a position paper that will be published in the proceeding of Bench 2018. Finally, for each benchmarking plan, a working group will be established, and each published position paper will act as a starting point for a formal benchmark specification, that will be released by BenchCouncil in future.
Proposals should provide the following information:
Proposals should be submitted in PDF format via e-mail to Dr. Chen Zheng (zhengchen@ict.ac.cn) and Dr. Zhen Jia (zhenj@cs.princeton.edu) with the subject "BenchCouncil Proposal". Submissions will be acknowledged via e-mail.
Important Dates
Submission deadline: Nov 10, 2018
Notification: Nov 13, 2018
Call for Papers
Topics of Interest
We solicit papers in all areas related to benchmarking, measuring, and optimizing systems. Short papers are encouraged, and the authors can extend their short papers to full versions after the conference. Topics of interest include (but are not limited to):
Big data domains
Artificial intelligence
NewSQL and distributed database systems
Blockchain
Datacenter, cloud and warehouse-scale computing
High performance computing
Mobile Robotics
Edge and fog computing
Big scientific data
Internet of Things (IoT)
Education, financial and power system
Submission Site
Bench 2018 Submission Site:
https://wi-lab.com/cyberchair/2018/bigdata18/scripts/submit.php?subarea=SP04&undisplay_detail=1&wh=/cyberchair/2018/bigdata18/scripts/ws_submit.php
Papers must be submitted in PDF. Short papers (no more than 4 pages in standard two-column IEEE conference format or 6 pages in LNCS format, not including references) are encouraged, and the authors can extend their short paper to 12 pages in LNCS format, not including references, in the final publication. If a full paper is submitted, the page limit is 8 pages in standard two-column IEEE conference format or 12 pages in LNCS format, not including references. The submissions will be judged based on the merit of the ideas rather than the length. After the conference, revised papers will be published by Springer LNCS (www.springer.com/lncs, indexed by EI). Please note that LNCS format is the final one for publishing.
LNCS Copyright Form for authors:
Consent to Publish
Formatting Instructions
IEEE conference format:
8.5" x 11" (DOC, PDF)
LaTex Formatting Macros
LNCS format:
LaTex Template
Please make sure your submission satisfies ALL of the following requirements:
Paper must be submitted in printable PDF format.
Please number the pages of your submission.
The submission must be formatted for black-and-white printers. Please make sure your figures are readable when printed in black-and-white.
The submission must describe unpublished work that is not currently under review of any other conference or journal venues.
Call for Tutorial
Tutorial proposals are solicited for Bench 2018, which will take place in Seattle, WA. Tutorials will be held on Dec 10-12.
Proposals for both half- and full-day tutorials are solicited on any topic that is relevant to the Bench 2018 audience. In our experiences, tutorials seeking to achieve either of the following goals have been particularly successful:
Describe an important piece of research infrastructure.
Educate the community on an emerging topic.
Title.
Presenter(s) and contact information.
Proposed duration (full day, half day).
1-2 paragraph abstract suitable for tutorial publicity.
1 paragraph biography per presenter suitable for tutorial publicity.
1-3 page description (for evaluation). This should include:
Submission Procedures
Proposals should provide the following information:
1. Tutorial scope and objectives,
2. Topics to be covered,
3. Target audience,
4. If the tutorial has been held previously, the location (i.e., conference), date, and number of attendees.
Proposals should be submitted in PDF format via e-mail to Lei Wang (wangle_2011@ict.ac.cn ) and Chen Zheng (zhengchen@ict.ac.cn) with the subject "Bench’18 Tutorial Proposal". Submissions will be acknowledged via e-mail.
Important Dates
Submission deadline: Oct 6, 2018
Notification: October 24th, 2018
Organization
| Program Chair | Prof. Jianfeng Zhan ICT, Chinese Academy of Sciences and University of Chinese Academy of Sciences |
| Program Committee |
Lizy K. John, The University of Texas at Austin USA Vijay Janapa Reddi, University of Texas at Austin USA and Google Wenguang Chen, Tsinghua University China Woongki Baek, UNIST Korea Yunji Chen, ICT, CAS China Piotr Luszczek, University of Tennessee USA Rui Hou, Institute of Information Engineering, CAS China Yueguo Chen, Renmin University of China H. Peter Hofstee, IBM USA Cheqing Jin, East China Normal University China Trevor E. Carlson, National University of Singapore (NUS) Singapore Junzhao Du, Xidian University China Hyogi Sim, Oak Ridge National Laboratory USA Weining Qian, East China Normal University China Zheng Cao, Alibaba China Matthias Nicola, IBM Germany Zhihui Du, Tsinghua University China Chunjie Luo, ICT, CAS Bin Ren, College of William and Mary USA Hua Chen, ICT, CAS China Jiaquan Gao, Nanjing Normal University China Jiannan Ouyang, Facebook USA Xueming Si, Fudan University China Yanjun Wu, Institute of Software Chinese Academy of Sciences China Lucas Mello Schnorr, Federal University of Rio Grande do Sul (UFRGS) Brazil Yunquan Zhang, ICT, CAS China Shengzhong Feng, Shenzhen Institutes of Advanced Technology, CAS China Zhen Jia, Princeton University USA Zhibin Yu, Shenzhen Institutes of Advanced Technology, CAS China Dapeng Wang, Institute of Software Chinese Academy of Sciences China Bo Wu, Colorado School of Mines USA Shaoliang Peng, National University of Defense Technology China Yong Qi, Xi'an jiaotong University China Gwangsun Kim, Arm Inc., Austin, Texas USA Rui Ren, ICT, CAS China Zujie Ren, Hangzhou Dianzi University China Xiaoyi Lu, The Ohio State University USA Lei Wang, ICT, CAS China Jianhui Li, Computer Network Information Center, CAS China Jianwu Wang, University of Maryland, Baltimore County USA Kenli Li, Hunan University China Suzanne Rivoire, Sonoma State University USA Rui Han, ICT, CAS China Jungang Xu, Chinese academy of sciences China Xu Liu, College of William and Mary USA Tong Wu, National Institute of Metrology, China Li ZHA, University of the Chinese Academy of Sciences Lei Liu, ICT, CAS China Chen Zheng, ICT, CAS China |
| Tutorial Co-Chairs | Dr. Lei Wang ICT, Chinese Academy of Sciences and University of Chinese Academy of Sciences Dr. Chen ZhengICT, Chinese Academy of Sciences and University of Chinese Academy of Sciences |
| Publication Chair | Prof. Xiaohui Hu Science & Technology on Integrated Information System Laboratory, Institute of Software Chinese Academy of Sciences |
| Web Chairs | Mr. Fei Tang University of Chinese Academy of Sciences |
| Publicity Chairs | Miss. Wanling Gao, University of Chinese Academy of Sciences
Miss. Rui Ren, ICT, Chinese Academy of Sciences Dr. Zhen Jia, Princeton University |
| Submission Co-Chairs | Miss. Rui Wang Science & Technology on Integrated Information System Laboratory, Institute of Software Chinese Academy of Sciences Miss. Ruiying Li Science & Technology on Integrated Information System Laboratory, Institute of Software Chinese Academy of Sciences |
| Sponsorship Chair | Dr. Biwei Xie ICT, Chinese Academy of Sciences |
Sponsors
Best Paper Awards
Previous Events
| Workshop | Dates | Location |
|---|---|---|
| BPOE-1 | October 7, 2013 | IEEE Big Data Conference, San Jose, CA |
| BPOE-2 | October 31, 2013 | CCF HPC China, Guilin, China |
| BPOE-3 | December 5,2013 | CCF Big Data Technology Conference 2013, BeiJing, China |
| BPOE-4 | March 1, 2014 | ASPLOS 2014, Salt Lake City, Utah, USA |
| BPOE-5 | September 5, 2014 | VLDB 2014, Hangzhou, Zhejiang Province, China |
| BPOE-6 | Spetember 4th, 2015 | VLDB 2015, Hilton, Waikoloa hotel·Kohala Coast, Hawai’i |
| BPOE-7 | April 3, 2016 | ASPLOS 2016, Atlanta, Georgia, USA |
| BPOE-8 | April 9, 2017 | ASPLOS 2017, Xi’an, China |
| BPOE-9 | March 25, 2018 | ASPLOS 2018, Williamsburg, VA, USA |
| SDBA-1 | June 12, 2018 | ICS 2018, Beijing, China |
1) J. Zhan, R. Han, C. Weng. Big Data Benchmarks, Performance Optimization, and Emerging Hardware, Springer LNCS, volume 8807, 2014.
2) J. Zhan, R. Han, R. V. Zicari. Big Data Benchmarks, Performance Optimization, and Emerging Hardware, Springer LNCS, volume 9495, 2016.
Location
Seattle, WA