BenchCouncil Transactions on Benchmarks, Standards and Evaluations

Open access

ISSN: 2772-4859

BenchCouncil Transactions on Benchmarks, Standards and Evaluations

Open access

BenchCouncil Transactions on Benchmarks, Standards and Evaluations (TBench) is an open-access journal dedicated to advancing the field of benchmarks, data sets, standards, evaluations and optimization...

BenchCouncil Transactions on Benchmarks, Standards and Evaluations (TBench) is an open-access journal dedicated to advancing the field of benchmarks, data sets, standards, evaluations and optimizations.

We invite submissions covering a wide range of topics from various disciplines, with a particular emphasis on interdisciplinary research. Whether it pertains to computers, AI, medicine, education, finance, business, psychology, or other social disciplines, all relevant contributions are welcome.

At TBench, we prioritize the reproducibility of research. We strongly encourage authors to ensure that their articles are prepared for open-source or artifact evaluation before submission.

Areas of interest include, but are not limited to:

  1. Problem definition as a benchmark
    • Position articles on the definitions of emerging or future problems/ challenges
    • Position articles on opening new research areas
    • Position articles on investigating the impact of new technologies on different disciplines
    • Research articles on instantiating new problem settings

  2. Evaluation standard as a benchmark
    • Definition, design, implementation, and validation of evaluation standards
    • Evaluation methodology and metrics
    • New abstractions and tools in evaluation
    • Simulation, emulation, and testbed methodologies and systems in evaluation

  3. Industry best practice as a benchmark
    • Searching and summarizing industry best practice
    • Evaluation and optimization of industry practice
    • Retrospective of industry practice
    • Characterizing and optimizing real-world applications and systems
    • Benchmarking

  4. State-of-the-art solution as a benchmark
    • State-of-the-art solutions to well-known benchmarks
    • Benchmarking state-of-the-art solutions
    • Preliminary but insightful solutions to new or emerging problems or benchmarks
    • Evaluations of state-of-the-art solutions in the real-world setting

  5. Data set as a benchmark
    • Explicit or implicit problem definition deduced from the data set
    • Detailed descriptions of research or industry datasets, including the methods used to collect the data and technical analyses supporting the quality of the measurements
    • Analyses or meta-analyses of existing data
    • Systems, technologies, and techniques that advance data sharing and reuse to support reproducible research
    • Tools that generate large-scale data while preserving their original characteristics
    • Evaluating the rigor and quality of the experiments used to generate the data and the completeness of the data description

  6. Workload characterization, evaluation, and retrospective of design/implementation of real-world:
    • Computer or AI applications or systems
    • Finance applications or systems
    • Education applications or systems
    • Business applications or systems
    • Medicine applications or systems
    • Other industry applications or systems

  7. Measurement and evaluation:
    • Instrumentation, sampling, tracing, and profiling of large-scale, real-world applications and systems
    • Collection and analysis of measurement data that yield new insights
    • Measurement-based modeling (e.g., workloads, scaling behavior, and assessment of performance bottlenecks)
    • Methods and tools to monitor and visualize measurement and evaluation data
    • Systems and algorithms that build on measurement-based findings
    • Advances in data collection, analysis, and storage, e.g., anonymization, querying, and sharing
    • Reappraisal of previous empirical measurements and measurement-based conclusions
    • Descriptions of challenges and future directions that the measurement and evaluation community should pursue

  8. Methodologies, metrics, abstractions, algorithms, and tools for:
    • Analytical modeling techniques and model validation
    • Workload characterization and benchmarking
    • Performance, scalability, power, and reliability analysis
    • Sustainability analysis and power management
    • System measurement, performance monitoring, and forecasting
    • Anomaly detection, problem diagnosis, and troubleshooting
    • Capacity planning, resource allocation, run-time management, and scheduling
    • Experimental design, statistical analysis, and simulation

Society affiliation

International Open Benchmark Council (BenchCouncil) is a non-profit organization that aims to benchmark, standardize, evaluate and incubate emerging technologies. Since its founding, BenchCouncil bears four fundamental responsibilities: establish unified benchmark science and engineering across multi-disciplines; define the problems or challenges ...

International Open Benchmark Council (BenchCouncil) is a non-profit organization that aims to benchmark, standardize, evaluate and incubate emerging technologies. Since its founding, BenchCouncil bears four fundamental responsibilities: establish unified benchmark science and engineering across multi-disciplines; define the problems or challenges of emerging and future computing; keep the benchmarks and standards community open, inclusive, and growing; and promote benchmark-based quantitative approaches to tackle multidisciplinary and interdisciplinary challenges. BenchCouncil also hosts a series of influential benchmark projects. BenchCouncil presents the achievement and rising star awards each year at its flagship Bench conference.

BinCodex: A comprehensive and multi-level dataset for evaluating binary code similarity detection techniques

TensorTable: Extending PyTorch for mixed relational and linear algebra pipelines

Evaluatology: The science and engineering of evaluation

View all ScienceDirect

1. A review of Blockchain Technology applications for financial services

2. Unlocking the opportunities through ChatGPT Tool towards ameliorating the education system

3. An era of ChatGPT as a significant futuristic support tool: A study on features, abilities, and challenges

View all ScienceDirect

News

BenchCouncil Transactions on Benchmarks, Standards and Evaluations is now indexed by Ei Compendex!

Learn more

BenchCouncil Transactions on Benchmarks, Standards and Evaluations is now indexed in Scopus

Learn more

BenchCouncil Transactions on Benchmarks, Standards and Evaluations is now Indexed in DOAJ

View all

Related journals

Stay Informed

Register your interest and receive email alerts tailored to your needs. Sign up below.