feat(agents): model-regression harness, spike-X cases and the first baseline by MarkusNeusinger · Pull Request #12116 · MarkusNeusinger/anyplot

MarkusNeusinger · 2026-10-10T01:07:18Z

Summary

Adds the model-regression harness (agents/evals/matrix.py): every selected eval case runs through the real /v1 flow in process (same plugins, run queue, judge, pipeline and gates as a user's request) with the models the flags name and the render backend of --renderer (remote, local, fake); per case it records status, attempts, gate failures, validator rejections, edit-apply failures, the reviewer's verdict, LLM calls, tokens by kind, model_version, cost at list price and latencies, writes a JSON report and a Markdown summary, diffs against the pinned model's baseline over the shared cases, and exits non-zero on a regression. A preflight render, an outage stop, --gcloud-token renewal, --budget-usd and a 30-render review page with Accept/Reject export make one command answer spike X and every later model or prompt comparison; report blind merges two arms into one unnamed gallery and report score reads the judgements back.
Adds the spike-X fixtures: 120 synthetic cases (10 specs from 10 plot families, eligible in matplotlib and seaborn, times six datasets: renamed headers, values times 10, 12 rows, 5,000 rows, DD.MM.YYYY and ISO dates, semicolon text with decimal commas) from the seeded make_fixtures.py, which a test keeps in sync with the committed files; plus agents/evals/eligibility.py, a model-free sweep that classifies every matplotlib and seaborn file of the catalogue; plus the first committed baseline, agents/evals/baselines/claude-haiku-5-5.json, from the spike-X run below.
The runtime now writes content-free attribution fields for every pipeline step (adapter outcome, validator rule ids, gate ids with render times, reviewer defect ids, tokens by kind, judge tokens) so the harness can price and count from the real flow; .dockerignore keeps agents/evals/ out of every image.

Spike X, Claude Haiku 5.5 arm (2026-10-10)

Run from a scratch merge of this branch with #12115 against the throwaway renderer of #12115, --provider anthropic-vertex --location eu --cases full --renderer remote --gcloud-token --budget-usd 4 --no-baseline --save-baseline. The baseline in this PR is this run.

Metric	Value
Runs	122, no harness or pipeline errors, no outage
Pass rate (host and probe gates)	83.6 % (102 of 122), 93 of them after the one repair round
Statuses	`ok` 14 (11.5 %), `needs_attention` 98, `failed` 10 (all `validation`)
Cost	$0.52 at list price for the matrix, $0.0051 per passed plot
End to end p50 / p95	18.5 s / 25.1 s (render p50 4.0 s, time to first event p50 0.5 s)
LLM calls per run	4.77

By spec: line-basic and line-timeseries 100 %, scatter-basic 92 %, area-basic and histogram-basic 92 %, box-basic 50 % (5 of 6 matplotlib runs end in validation: the adapter replaces the df = load_user_data() placeholder, rule placeholder-count), pie-basic 58 %. By perturbation: 12 rows weakest at 70 %, values times 10 best at 95 %. matplotlib 78.7 %, seaborn 88.5 %.

What keeps a passed run at needs_attention instead of ok, in order: the probe gate G3 (text beyond the canvas) fails in 60 runs and reports in 56 more, and the reviewer's AR-09 lines agree in most of them; the reviewer's answer could not be read in 18 runs; the adapter's answer missed its schema in 23 of 141 calls, each costing the repair round; the IMPRINT palette rule (VQ-07) stands in 11 residual lines. These are the first items for the prompt and validator work on the Claude arm, not for this PR. The exit criterion of phase 0 asks for at least 80 % passing after at most one repair (met) and the owner's acceptance of at least 75 % on the 30-render sample (open: the gallery is with the owner).

Eligibility sweep (no model call, about 9 s): matplotlib 285 of 325 files eligible, seaborn 287 of 325; blocked by SECURITY findings 29 / 29, the map specs 13 / 13, a missing THEME block 16 / 11, a savefig target other than f"plot-{THEME}.png" 16 / 11.

Plan

Design: docs/concepts/agent-network.md ("Model-version regression harness", spike X in "Phase 0"). The 120-case matrix needs the remote backend of #12115 to render on Cloud Run; the harness itself merges independently (its tests use scripted models and the fake renderer).

Test plan

uv run ruff check . && ruff format --check ., mypy api core agents, pytest tests/unit tests/integration (7888 passed before the baseline; with it the baseline test runs: tests/unit/agents/evals 81 passed), changelog check, uv lock --check
Eligibility sweep on the current catalogue (numbers above)
Spike X on the Claude arm through the throwaway renderer (numbers above); the committed baseline is that run
Owner: open the review page (gallery.html of the run, 30 renders that passed the gates), judge each one, export the judgements; the acceptance rate is the phase-0 criterion (75 %)
The Gemini arm (--provider gemini --budget-usd 18) and the blind two-arm gallery wait for the owner's go: about $10 to $15 at list price

agents/evals/matrix.py runs eval cases through the real /v1 flow in process (httpx ASGITransport against agents.main:app, the same plugins, run queue, judge, pipeline and gates) with the models the flags name and the render backend --renderer names (remote, local, fake). Per run it records status, attempts, gate failures by gate, validator rejections, edit-apply failures, the reviewer verdict, LLM calls, tokens by kind, model_version, list-price cost and latencies; writes a JSON report, a Markdown summary and a blind review gallery (30 sampled renders, accept/reject pairs, JSON export); diffs against the pinned model's baseline and exits 1 on a pass-rate regression, 3 when --budget-usd stops it. agents/evals/make_fixtures.py writes 120 seeded synthetic cases: 10 specs of 10 plot families (each clean or coupled in matplotlib and seaborn) times six datasets (renamed, x10, n12, n5000, date, decimal-comma). The two hand-written cases stay and join the 12-case smoke set. The pipeline, the Budget plugin and the judges now write content-free attribution fields (gate ids, validator rule ids, adapter outcomes, the reviewer's defect ids, token counts by kind, judge input and output tokens) that the harness reads per request id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M

- Preflight render of each library's catalogue file before the first case (exit 2 when the renderer cannot render); a pipeline that ended in RendererUnavailable, or three errors in a row, stop the run as an outage (exit 4). pipeline_result now carries the exception class of an error. - --gcloud-token mints the renderer ID token with the gcloud CLI and renews it before its one-hour expiry; remote and local renders run as ENVIRONMENT=development outside Cloud Run. - The baseline diff compares the cases both reports hold; harness errors count as failures and exit 1; --save-baseline refuses runs with errors. - The budget stops before a case that could pass it. - The gallery samples passed renders only; `report blind` merges two runs into one blind gallery and `report score` gives acceptance per run. - agents/evals/eligibility.py: the model-free catalogue eligibility sweep. - .dockerignore excludes agents/evals; docs, README and fragment updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M

122 cases on 2026-10-10 through the throwaway renderer: 102 passed the gates (83.6 %), 14 ok, 98 needs_attention, 10 failed (validation), $0.52 at list price, end to end p50 18.5 s. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M

… fragment Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M

Copilot

🟡 Changes recommended

Service failures and missing render artifacts can evade outage detection and contaminate regression baselines.

7 open findings

What changed in this PR

Adds a production-path model-regression harness for evaluating agent models, prompts, rendering quality, reliability, latency, and cost.

Changes:

Adds matrix execution, reporting, galleries, pricing, eligibility scanning, and a committed Claude baseline.
Adds 120 reproducible spike-X fixture cases and comprehensive tests.
Adds content-free runtime attribution and excludes evaluation data from service images.

File	Description
`.dockerignore`	Excludes evaluation assets from images.
`.gitignore`	Ignores local evaluation outputs.
`agents/README.md`	Documents harness operation and spike X.
`agents/anyplot/models.py`	Reports judge token splits.
`agents/anyplot/pipeline.py`	Emits pipeline attribution.
`agents/anyplot/plugins/budget.py`	Emits token-kind attribution.
`agents/anyplot/plugins/ledger.py`	Defines attribution token fields.
`agents/anyplot/plugins/scope_guard.py`	Reports judge token splits.
`agents/anyplot/render/gates.py`	Records failed gate identifiers.
`agents/evals/baselines/README.md`	Documents baseline management.
`agents/evals/baselines/claude-haiku-5-5.json`	Adds the initial Claude baseline.
`agents/evals/cases.py`	Loads and selects evaluation cases.
`agents/evals/eligibility.py`	Adds catalogue eligibility scanning.
`agents/evals/fixtures/README.md`	Documents fixture format and provenance.
`agents/evals/fixtures/cases/area-basic-matplotlib-date/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/area-basic-matplotlib-date/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/area-basic-matplotlib-decimal-comma/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/area-basic-matplotlib-decimal-comma/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/area-basic-matplotlib-n12/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/area-basic-matplotlib-n12/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/area-basic-matplotlib-n5000/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/area-basic-matplotlib-n5000/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/area-basic-matplotlib-renamed/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/area-basic-matplotlib-renamed/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/area-basic-matplotlib-x10/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/area-basic-matplotlib-x10/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/area-basic-seaborn-date/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/area-basic-seaborn-date/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/area-basic-seaborn-decimal-comma/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/area-basic-seaborn-decimal-comma/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/area-basic-seaborn-n12/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/area-basic-seaborn-n12/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/area-basic-seaborn-n5000/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/area-basic-seaborn-n5000/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/area-basic-seaborn-renamed/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/area-basic-seaborn-renamed/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/area-basic-seaborn-x10/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/area-basic-seaborn-x10/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/bar-grouped-matplotlib-date/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/bar-grouped-matplotlib-date/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/bar-grouped-matplotlib-decimal-comma/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/bar-grouped-matplotlib-decimal-comma/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/bar-grouped-matplotlib-n12/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/bar-grouped-matplotlib-n12/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/bar-grouped-matplotlib-n5000/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/bar-grouped-matplotlib-n5000/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/bar-grouped-matplotlib-renamed/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/bar-grouped-matplotlib-renamed/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/bar-grouped-matplotlib-x10/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/bar-grouped-matplotlib-x10/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/bar-grouped-seaborn-date/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/bar-grouped-seaborn-date/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/bar-grouped-seaborn-decimal-comma/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/bar-grouped-seaborn-decimal-comma/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/bar-grouped-seaborn-n12/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/bar-grouped-seaborn-n12/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/bar-grouped-seaborn-n5000/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/bar-grouped-seaborn-n5000/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/bar-grouped-seaborn-renamed/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/bar-grouped-seaborn-renamed/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/bar-grouped-seaborn-x10/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/bar-grouped-seaborn-x10/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/bar-grouped-seaborn/case.json`	Updates the handwritten smoke case.
`agents/evals/fixtures/cases/box-basic-matplotlib-date/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/box-basic-matplotlib-date/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/box-basic-matplotlib-decimal-comma/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/box-basic-matplotlib-decimal-comma/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/box-basic-matplotlib-n12/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/box-basic-matplotlib-n12/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/box-basic-matplotlib-n5000/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/box-basic-matplotlib-n5000/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/box-basic-matplotlib-renamed/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/box-basic-matplotlib-renamed/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/box-basic-matplotlib-x10/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/box-basic-matplotlib-x10/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/box-basic-seaborn-date/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/box-basic-seaborn-date/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/box-basic-seaborn-decimal-comma/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/box-basic-seaborn-decimal-comma/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/box-basic-seaborn-n12/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/box-basic-seaborn-n12/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/box-basic-seaborn-n5000/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/box-basic-seaborn-n5000/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/box-basic-seaborn-renamed/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/box-basic-seaborn-renamed/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/box-basic-seaborn-x10/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/box-basic-seaborn-x10/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/heatmap-basic-matplotlib-date/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/heatmap-basic-matplotlib-date/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/heatmap-basic-matplotlib-decimal-comma/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/heatmap-basic-matplotlib-decimal-comma/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/heatmap-basic-matplotlib-n12/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/heatmap-basic-matplotlib-n12/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/heatmap-basic-matplotlib-n5000/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/heatmap-basic-matplotlib-n5000/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/heatmap-basic-matplotlib-renamed/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/heatmap-basic-matplotlib-renamed/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/heatmap-basic-matplotlib-x10/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/heatmap-basic-matplotlib-x10/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/heatmap-basic-seaborn-date/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/heatmap-basic-seaborn-date/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/heatmap-basic-seaborn-decimal-comma/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/heatmap-basic-seaborn-decimal-comma/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/heatmap-basic-seaborn-n12/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/heatmap-basic-seaborn-n12/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/heatmap-basic-seaborn-n5000/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/heatmap-basic-seaborn-n5000/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/heatmap-basic-seaborn-renamed/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/heatmap-basic-seaborn-renamed/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/heatmap-basic-seaborn-x10/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/heatmap-basic-seaborn-x10/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/histogram-basic-matplotlib-date/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/histogram-basic-matplotlib-date/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/histogram-basic-matplotlib-decimal-comma/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/histogram-basic-matplotlib-decimal-comma/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/histogram-basic-matplotlib-n12/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/histogram-basic-matplotlib-n12/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/histogram-basic-matplotlib-n5000/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/histogram-basic-matplotlib-n5000/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/histogram-basic-matplotlib-renamed/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/histogram-basic-matplotlib-renamed/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/histogram-basic-matplotlib-x10/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/histogram-basic-matplotlib-x10/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/histogram-basic-seaborn-date/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/histogram-basic-seaborn-date/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/histogram-basic-seaborn-decimal-comma/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/histogram-basic-seaborn-decimal-comma/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/histogram-basic-seaborn-n12/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/histogram-basic-seaborn-n12/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/histogram-basic-seaborn-n5000/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/histogram-basic-seaborn-n5000/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/histogram-basic-seaborn-renamed/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/histogram-basic-seaborn-renamed/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/histogram-basic-seaborn-x10/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/histogram-basic-seaborn-x10/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/line-basic-matplotlib-date/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/line-basic-matplotlib-date/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/line-basic-matplotlib-decimal-comma/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/line-basic-matplotlib-decimal-comma/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/line-basic-matplotlib-n12/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/line-basic-matplotlib-n12/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/line-basic-matplotlib-n5000/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/line-basic-matplotlib-n5000/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/line-basic-matplotlib-renamed/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/line-basic-matplotlib-renamed/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/line-basic-matplotlib-x10/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/line-basic-matplotlib-x10/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/line-basic-seaborn-date/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/line-basic-seaborn-date/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/line-basic-seaborn-decimal-comma/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/line-basic-seaborn-decimal-comma/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/line-basic-seaborn-n12/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/line-basic-seaborn-n12/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/line-basic-seaborn-n5000/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/line-basic-seaborn-n5000/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/line-basic-seaborn-renamed/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/line-basic-seaborn-renamed/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/line-basic-seaborn-x10/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/line-basic-seaborn-x10/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/line-timeseries-matplotlib-date/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/line-timeseries-matplotlib-date/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/line-timeseries-matplotlib-decimal-comma/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/line-timeseries-matplotlib-decimal-comma/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/line-timeseries-matplotlib-n12/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/line-timeseries-matplotlib-n12/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/line-timeseries-matplotlib-n5000/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/line-timeseries-matplotlib-n5000/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/line-timeseries-matplotlib-renamed/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/line-timeseries-matplotlib-renamed/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/line-timeseries-matplotlib-x10/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/line-timeseries-matplotlib-x10/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/line-timeseries-seaborn-date/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/line-timeseries-seaborn-date/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/line-timeseries-seaborn-decimal-comma/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/line-timeseries-seaborn-decimal-comma/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/line-timeseries-seaborn-n12/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/line-timeseries-seaborn-n12/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/line-timeseries-seaborn-n5000/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/line-timeseries-seaborn-n5000/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/line-timeseries-seaborn-renamed/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/line-timeseries-seaborn-renamed/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/line-timeseries-seaborn-x10/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/line-timeseries-seaborn-x10/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/pie-basic-matplotlib-date/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/pie-basic-matplotlib-date/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/pie-basic-matplotlib-decimal-comma/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/pie-basic-matplotlib-decimal-comma/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/pie-basic-matplotlib-n12/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/pie-basic-matplotlib-n12/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/pie-basic-matplotlib-n5000/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/pie-basic-matplotlib-n5000/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/pie-basic-matplotlib-renamed/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/pie-basic-matplotlib-renamed/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/pie-basic-matplotlib-x10/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/pie-basic-matplotlib-x10/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/pie-basic-seaborn-date/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/pie-basic-seaborn-date/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/pie-basic-seaborn-decimal-comma/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/pie-basic-seaborn-decimal-comma/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/pie-basic-seaborn-n12/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/pie-basic-seaborn-n12/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/pie-basic-seaborn-n5000/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/pie-basic-seaborn-n5000/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/pie-basic-seaborn-renamed/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/pie-basic-seaborn-renamed/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/pie-basic-seaborn-x10/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/pie-basic-seaborn-x10/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/scatter-basic-matplotlib-date/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/scatter-basic-matplotlib-date/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/scatter-basic-matplotlib-decimal-comma/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/scatter-basic-matplotlib-decimal-comma/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/scatter-basic-matplotlib-n12/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/scatter-basic-matplotlib-n12/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/scatter-basic-matplotlib-n5000/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/scatter-basic-matplotlib-n5000/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/scatter-basic-matplotlib-renamed/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/scatter-basic-matplotlib-renamed/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/scatter-basic-matplotlib-x10/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/scatter-basic-matplotlib-x10/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/scatter-basic-matplotlib/case.json`	Updates the handwritten smoke case.
`agents/evals/fixtures/cases/scatter-basic-seaborn-date/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/scatter-basic-seaborn-date/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/scatter-basic-seaborn-decimal-comma/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/scatter-basic-seaborn-decimal-comma/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/scatter-basic-seaborn-n12/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/scatter-basic-seaborn-n12/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/scatter-basic-seaborn-n5000/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/scatter-basic-seaborn-n5000/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/scatter-basic-seaborn-renamed/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/scatter-basic-seaborn-renamed/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/scatter-basic-seaborn-x10/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/scatter-basic-seaborn-x10/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/violin-basic-matplotlib-date/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/violin-basic-matplotlib-date/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/violin-basic-matplotlib-decimal-comma/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/violin-basic-matplotlib-decimal-comma/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/violin-basic-matplotlib-n12/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/violin-basic-matplotlib-n12/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/violin-basic-matplotlib-n5000/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/violin-basic-matplotlib-n5000/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/violin-basic-matplotlib-renamed/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/violin-basic-matplotlib-renamed/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/violin-basic-matplotlib-x10/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/violin-basic-matplotlib-x10/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/violin-basic-seaborn-date/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/violin-basic-seaborn-date/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/violin-basic-seaborn-decimal-comma/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/violin-basic-seaborn-decimal-comma/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/violin-basic-seaborn-n12/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/violin-basic-seaborn-n12/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/violin-basic-seaborn-n5000/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/violin-basic-seaborn-n5000/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/violin-basic-seaborn-renamed/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/violin-basic-seaborn-renamed/data.csv`	Synthetic fixture data.
`agents/evals/fixtures/cases/violin-basic-seaborn-x10/case.json`	Eval case metadata.
`agents/evals/fixtures/cases/violin-basic-seaborn-x10/data.csv`	Synthetic fixture data.
`agents/evals/make_fixtures.py`	Generates deterministic spike-X cases.
`agents/evals/matrix.py`	Implements matrix execution and baselines.
`agents/evals/pricing.py`	Calculates model costs.
`agents/evals/report.py`	Builds summaries, diffs, and galleries.
`agents/main.py`	Adds dataset-judge attribution.
`changelog.d/agents-evals.md`	Records the feature changes.
`docs/concepts/agent-network.md`	Updates the agent design status.
`docs/reference/repository.md`	Documents the evaluation directory.
`tests/unit/agents/evals/__init__.py`	Adds the eval test package.
`tests/unit/agents/evals/conftest.py`	Shares runtime fixtures and HTML checks.
`tests/unit/agents/evals/test_baselines.py`	Tests committed baseline integrity.
`tests/unit/agents/evals/test_cases.py`	Tests case loading and selection.
`tests/unit/agents/evals/test_eligibility.py`	Tests eligibility reporting.
`tests/unit/agents/evals/test_make_fixtures.py`	Tests deterministic fixture generation.
`tests/unit/agents/evals/test_matrix.py`	Tests harness execution and safeguards.
`tests/unit/agents/evals/test_report.py`	Tests reports, diffs, and galleries.
`tests/unit/agents/runtime/test_models.py`	Tests judge token splits.
`tests/unit/agents/runtime/test_render.py`	Tests failed-gate attribution.

🧠 Review effort: Balanced

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

codecov · 2026-10-10T01:21:42Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

… as the case's outcome Copilot's findings on #12116: - A dataset or bindings answer other than a refusal on the merits (403, 409, 413, 422) is the service's error: the judge's 503 guard_unavailable now ends the run with status error, so an outage trips the three-errors stop and keeps the run out of a baseline. - A shipped plot without its PNG, or one whose artifact cannot be fetched, is an error (reason no_png) rather than a pass the galleries cannot show. - is_error also counts status error (a session that could not open, a turn refused with capacity, no result), an error event that came with a plot, and a recorded pipeline error. - The README, the baselines README, the design doc's status line and the baseline test describe the committed Claude Haiku 5.5 baseline; the test no longer skips. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M

# Conflicts: # agents/README.md # docs/concepts/agent-network.md # docs/reference/repository.md

MarkusNeusinger and others added 3 commits October 10, 2026 01:25

Copilot AI balanced review requested due to automatic review settings October 10, 2026 01:07

Copilot started reviewing on behalf of MarkusNeusinger October 10, 2026 01:07 View session

MarkusNeusinger and others added 2 commits October 10, 2026 03:08

chore(changelog): reference #12116 and name the first baseline in the…

b104cdf

… fragment Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M

docs(agents): record the first spike-X measurements in the design doc

5338b78

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M

Copilot AI reviewed Oct 10, 2026

View reviewed changes

MarkusNeusinger and others added 3 commits October 10, 2026 03:22

Merge remote-tracking branch 'origin/main' into feat/agents-evals

e05ac66

Merge remote-tracking branch 'origin/main' into feat/agents-evals

0597294

# Conflicts: # agents/README.md # docs/concepts/agent-network.md # docs/reference/repository.md

MarkusNeusinger merged commit 77a7811 into main Oct 10, 2026
11 checks passed

MarkusNeusinger deleted the feat/agents-evals branch October 10, 2026 07:20

Navigation Menu

Repository navigation

feat(agents): model-regression harness, spike-X cases and the first baseline - #12116

feat(agents): model-regression harness, spike-X cases and the first baseline#12116
MarkusNeusinger merged 8 commits into
mainfrom
feat/agents-evals

MarkusNeusinger commented Oct 10, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Oct 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

MarkusNeusinger commented Oct 10, 2026

Summary

Spike X, Claude Haiku 5.5 arm (2026-10-10)

Plan

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Oct 10, 2026

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants