Evaluating decisions in data workflows.
500 tasks · 6 subsets · English + Chinese · v0.5
Design · Results · Quick Start · Contributing · Citation
DataJevBench evaluates decisions within data workflows: which records to keep, which processing steps to choose, how to judge output quality, whether evidence supports an analysis, and what action to take next.
Developed by the DataJuicer Team, the benchmark compares models on shared task inputs and scoring rules to identify their strengths and limitations across data decisions. All 500 tasks and reference answers are openly available.
Note
Supported in part by Data-Juicer. Parts of the evaluation set are built using its data-processing operators and documentation, connecting benchmark decisions to practical data workflows.
Data quality depends on choices throughout a workflow. Filtering determines which material enters a dataset. Processing plans determine how that material is transformed. Quality checks and evidence assessment determine which outputs are useful and which conclusions are supported. Workflow control connects these decisions to the next action.
These choices require a model to combine the available material with explicit rules, quality criteria and the current workflow state. The six subsets below cover decisions from data admission through output validation.
| Subset | Tasks | Share | Central question |
|---|---|---|---|
| D1 · Data filtering | 90 | 18% | Should this record or batch be admitted? |
| D2 · Processing plan selection | 90 | 18% | Which available operations satisfy the processing requirements? |
| D3 · Output quality assessment | 80 | 16% | How well does this output meet the quality rubric? |
| D4 · Information sufficiency | 100 | 20% | Does the available material determine the requested answer? |
| D5 · Evidence and privacy | 80 | 16% | Is a claim supported, and does a disclosure meet the privacy rules? |
| D6 · Workflow control | 60 | 12% | Which action fits the current workflow state? |
The illustration and examples below show simplified scenarios for these six decision types.
Each task presents three elements:
| Element | Content |
|---|---|
| State | The material to reason over, such as records, documents, tables, operator descriptions or workflow status. |
| Instruction | The decision to make, together with the relevant rules or quality criteria. |
| Candidates | Named answer options or ordered rubric levels from which the model selects an answer. |
The task set contains 480 choice questions (96%) and 20 ordinal score questions (4%). Choice tasks include binary decisions, comparisons and selection among several actions or plans. Score tasks ask for a level on the supplied ordered rubric.
Instructions are 300 English (60%) and 200 Chinese (40%). Task materials span financial reports, scientific papers, descriptions and summaries, processing operators, and workflow states.
| Metric | What it describes |
|---|---|
| Accuracy ↑ | Correct answers divided by 500, with equal weight for each task. |
| D1–D6 Accuracy ↑ | Correct answers divided by the task count within each subset. |
| ECE10 ↓ | Agreement between confidence and observed accuracy across ten confidence bins. |
| Brier ↓ | Average sum of squared errors between predicted probabilities and the correct-answer indicator. |
The overall score summarizes performance across the full task distribution. Subset scores describe performance on individual capabilities, such as selecting a processing plan or checking evidence. Missing and invalid answers contribute zero to Accuracy. ECE10 and Brier are available when all 500 tasks have valid probability outputs.
v0.5 uses a probability-sum tolerance of 1e-3. Accepted probabilities are scored as returned.
Results cover 500 tasks per model, using each model's recorded inference configuration and the v0.5 scorer. Models are ordered by overall Accuracy, with shared ranks for equal scores.
Evaluation snapshot: 2026-10-28
| Rank | Model | Accuracy ↑ | Valid answers | ECE10 ↓ | Brier ↓ |
|---|---|---|---|---|---|
| 1 | vLLM-SR d3-ultra | 83.4% | 500/500 | 0.0669 | 0.2405 |
| 2 | AutoTrust-JEV-27B v1.1 | 82.4% | 500/500 | 0.0465 | 0.2341 |
| 2 | GLiDE | 82.4% | 500/500 | 0.0309 | 0.2404 |
| 4 | vLLM-SR d3 (27B) | 81.6% | 500/500 | 0.0311 | 0.2477 |
| 5 | Mammoth-1 | 80.6% | 500/500 | 0.0483 | 0.2744 |
| 6 | Perplexity Decider v1.2 (27B) | 80.2% | 500/500 | 0.0431 | 0.2783 |
| 7 | StartLux Decision v1.1 27B Instant | 78.8% | 500/500 | 0.0442 | 0.2592 |
| 8 | Mercury Decide (Inception) | 78.4% | 500/500 | 0.1416 | 0.3600 |
| 9 | Jeffan-31B-1710 | 74.4% | 500/500 | 0.0827 | 0.3731 |
| 10 | Jev 1.14 | 73.8% | 476/500 | — | — |
| 11 | Cloudflare Clef (27B) | 73.6% | 500/500 | 0.0787 | 0.3670 |
| 12 | Microsoft Decision-1 | 72.6% | 500/500 | 0.0948 | 0.3797 |
| 13 | GPT-6 Luna Decisions | 71.2% | 500/500 | 0.1047 | 0.4162 |
| 14 | Strands Decider 26B-A4B (Gemma 4) | 60.6% | 500/500 | 0.0779 | 0.4873 |
Valid answers report coverage out of 500. A dash marks a calibration metric that requires complete probability coverage. The result snapshot also records D1–D6 scores, model settings, revisions and evidence hashes.
Python 3.10+ suffices for verification, HTTP evaluation and scoring. Models use their own inference environment. From the repository root:
python -m pip install -e .
python -m datajevbench doctor \
--engine datajevbench.engines:create_http \
--engine-options configs/systemone.example.json --model-id your-modeldoctor checks configuration without calling a model. Tasks and reference answers are in benchmark/records.jsonl, with gold-free requests in benchmark/inputs.jsonl. Verify the included public data:
python -m datajevbench verifyBefore inference, configure a model service and local JSON files using the deployment guide, then follow the full run commands. HTTP uses /v1/systemone; a Python module:factory adapter is also supported. Use --limit 2 for an initial diagnostic and a separate output directory for a full run.
Existing model implementations exposing predict(state, questions) can use the predict bridge, keeping model runtimes and their dependencies in the model environment.
A run saves environment.json and results.jsonl. Score an existing run with:
python -m datajevbench score --run runs/your-model-v05Scoring writes scores.json without model calls. run --resume retries pending and failed tasks under matching assets and configuration, retaining all attempts.
Open an Issue with the model/checkpoint, scores.json, inference settings and code revision when contributing results. For code changes, run:
PYTHONDONTWRITEBYTECODE=1 python scripts/check_repo.pyresults/leaderboard.json is the single source for both README tables and the site. Historical per-task predictions are not bundled; checking snapshot consistency is not historical score replay. Existing scores retain their original source-data hashes; the public ID conversion did not rerun models. See reproducibility.
Benchmark data uses MIT for project-authored contributions; third-party source terms and attribution remain in the source notices. Repository-specific code, documentation and illustrations still need an owner-selected license; the data license does not automatically cover them. Attribution is retained in NOTICE. All 500 tasks and reference answers are included. Data format documents the files and hash checks; publication notes record the original-to-public ID and metadata projection.
- Benchmark: Task design · Data directory · v0.5 metadata
- Evaluation: Quick start · Model deployment · Scoring protocol
- Results and releases: Result data · Reproducibility · Source manifest · Site preview
@misc{datajevbench,
title = {DataJevBench: Evaluating Decisions in Data Workflows},
author = {{Data-Juicer Team}},
year = {2026},
url = {https://github.com/datajuicer/DataJevBench},
note = {Benchmark v0.5}
}