Aims and Scopes


BenchCouncil Transactions on Benchmarks, Standards, and Evaluations (TBench) publishes position articles that open new research areas, research articles that address new problems, methodologies, tools, survey articles that build up comprehensive knowledge, and comments articles that argue the published articles. The submissions should deal with the benchmarks, standards, and evaluation research areas. Particular areas of interest include, but are not limited to:

  • 1. Generalized benchmark science and engineering (see https://www.sciencedirect.com/science/article/pii/S2772485921000120), including but not limited to
    • measurement standards
    • standardized data sets with defined properties
    • representative workloads
    • representative data sets
    • best practices
  • 2. Benchmark and standard specifications, implementations, and validations of:
    • Big Data
    • AI
    • HPC
    • Machine learning
    • Big scientific data
    • Datacenter
    • Cloud
    • Warehouse-scale computing
    • Mobile robotics
    • Edge and fog computing
    • IoT
    • Chain block
    • Data management and storage
    • Financial domains
    • Education domains
    • Medical domains
    • Other application domains
  • 3. Data sets
    • Detailed descriptions of research or industry datasets, including the methods used to collect the data and technical analyses supporting the quality of the measurements.
    • Analyses or meta-analyses of existing data and original articles on systems, technologies, and techniques that advance data sharing and reuse to support reproducible research.
    • Evaluating the rigor and quality of the experiments used to generate the data and the completeness of the data description.
    • Tools generating large-scale data while preserving their original characteristics.
  • 4. Workload characterization, quantitative measurement, design, and evaluation studies of:
    • Computer and communication networks, protocols, and algorithms
    • Wireless, mobile, ad-hoc and sensor networks, IoT applications
    • Computer architectures, hardware accelerators, multi-core processors, memory systems, and storage networks
    • High-Performance Computing
    • Operating systems, file systems, and databases
    • Virtualization, data centers, distributed and cloud computing, fog, and edge computing
    • Mobile and personal computing systems
    • Energy-efficient computing systems
    • Real-time and fault-tolerant systems
    • Security and privacy of computing and networked systems
    • Software systems and services, and enterprise applications
    • Social networks, multimedia systems, Web services
    • Cyber-physical systems, including the smart grid
  • 5. Methodologies, metrics, abstractions, algorithms, and tools for:
    • Analytical modeling techniques and model validation
    • Workload characterization and benchmarking
    • Performance, scalability, power, and reliability analysis
    • Sustainability analysis and power management
    • System measurement, performance monitoring, and forecasting
    • Anomaly detection, problem diagnosis, and troubleshooting
    • Capacity planning, resource allocation, run time management, and scheduling
    • Experimental design, statistical analysis, simulation
  • 6. Measurement and evaluation
    • Evaluation methodology and metric
    • Testbed methodologies and systems
    • Instrumentation, sampling, tracing, and profiling of Large-scale real-world applications and systems
    • Collection and analysis of measurement data that yield new insights
    • Measurement-based modeling (e.g., workloads, scaling behavior, assessment of performance bottlenecks)
    • Methods and tools to monitor and visualize measurement and evaluation data
    • Systems and algorithms that build on measurement-based findings
    • Advances in data collection, analysis, and storage (e.g., anonymization, querying, sharing)
    • Reappraisal of previous empirical measurements and measurement-based conclusions
    • Descriptions of challenges and future directions the measurement and evaluation community should pursue