Skip navigation

BenchCouncil: International Open Benchmarking Council

 

BenchCouncil's Mission

International Open Benchmarking Council (BenchCouncil) is a non-profit benchmarking organization, which aims to promote multi-disciplinary benchmarking research and practice and foster collaboration and interaction between industry and academia.


Current and planned activities of the BenchCouncil include:

  • Establish and maintain a repository of benchmark specifications for quantitative system and algorithm evaluation and analysis.

  • Review, shepherd, and release open-source benchmark implementations.

  • Publish newsletters and research articles in the area of benchmarking.

  • Organize conferences, workshops, and teleconferences fostering the transfer of knowledge between industry and academia in the areas of benchmarking.

  • Organize challenges and competition using released benchmarks.

BenchCouncil publishes a journal: BenchCouncil Transaction on Benchmarking, measuring, and Optimizing (in short, TBench).

BenchCouncil organizes BenchCouncil main conference (Bench) and BenchCouncil annual System Technology conference (BenchCouncil ATC).


BenchCouncil's Philosophy

Open, Transparent, and Balanced

Open: anyone can submit a benchmark proposal. Anyone can contribute to benchmark specification and implementations.

Transparent: any outcome will be published by TBench. Each one’s contribution will be recorded and publicly available.

Balanced: each working group has to consist of half industry researchers and half academia researchers.


BenchCouncil's Procedure

Each benchmarking pipeline consists of the following six steps: benchmark proposal, formation of working group, publish benchmark specifications, open-source implementations of benchmark specifications, challenges and competitions, and archive performance numbers.



  • Benchmark proposal. BenchCouncil encourages any company, institute, organization or individual to submit a benchmark proposal. Once a proposal is accepted, the authors will be invited to give a talk at BenchCouncil main conference or follow-up workshops, and finally write a position paper on TBench. On the same theme, different groups are encouraged to submit their proposals.

  • Formation of working groups. For each benchmark, a working group will be formed. Two leaders will be designated, one from industry and another from academia. The working group has the responsibilities to seek funding to support the research and development of benchmarks. Generally, a series of discussions will be held at BenchCouncil main conferences, follow-up workshops and teleconference.

  • Publish benchmark specifications. BenchCouncil encourages publishing benchmark specification on TBench on time. Comments to the specifications and reply to the comments will also be published on TBench.

  • Open-source implementations of benchmark specifications. BenchCouncil encourages anyone to contribute to the open-source implementations of benchmarks specifications. The practice reports of open-source implementations are encouraged to publish on TBench.

  • Challenges and competitions. BenchCouncil will organize challenges and competitions using the released benchmarks.

  • Archive performance numbers. The performance numbers will be archived on TBench and the BenchCouncil web site.