Skip to content

feat(runs): read a task's repeated attempts together, by pass rate, with pass@k and pass^k - #146

Merged
aitianshi merged 6 commits into
mainfrom
feat/compare-pass-count-sorting
Nov 17, 2026
Merged

aitianshi merged 6 commits into
mainfrom
feat/compare-pass-count-sorting

Conversation

@aitianshi

@aitianshi aitianshi commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #130
Closes #145

What

An attempt is one run of a task, counted whatever its result (Harbor's n_attempts, the standard's --repeat); a retry is a further try of an attempt after an infrastructure or agent failure, and only its last try is kept as the attempt's trial. A task's attempts in one job are now read together, by one backend module (nullius.runs.attempts), the way the run's adjusted rate reads trials: an infra attempt is counted apart and left out of the pass count, the means and pass@k, and an agent's timeout or crash counts as a failure. Attempts are numbered by the run's own number when its trials carry one (the standard's u0000r2), so an attempt retried after the others keeps its place, else by start time. The trial page gets a tab per attempt and an All attempts table. The job page shows one row per repeated task (its pass count, a square per attempt, the instruction score once, the mean tokens and times) that opens onto its attempts, and the run page shows pass@k and pass^k. Compare (#130) judges a task by its pass rate over every attempt instead of its last one, ranks the rows by how far that rate moved (read with one pass and one fail added, so a single flipped attempt ranks below a steady gap), shows pass@k and pass^k in its run summary, and gives each side of the side-by-side page its own attempt tabs. The standard's pass_hat calls the same estimators; what it writes is unchanged.

Touches

backend · routes · templates and static · docs and site · the standard (pass_hat calls the shared estimators)

Before and after

Before After
Job page: three rows named 0001, one per attempt Job page: one row for 0001, 2 of 3 passed, opened onto attempts 1 to 3 with the means on the task row
Run page rate block without pass@k Run page rate block with pass@k and pass^k for k 1 to 3
Trial page with no attempt tabs Trial page with 2/3 passed, a tab per attempt and All attempts
Compare run summary ending without pass@k Compare run summary with pass@1 to pass@3 and pass^1 to pass^3
Compare task row: 3 attempts, last indexed shown Compare task row: 2/3 passed against 0/3 passed

An infra attempt counted apart, after (a hand-made run in Harbor's layout): t1 reads 1/2 · 1 infra and its means leave the rate-limited attempt out; t2's timeout counts as a failure.

Job page: t1 at 1/2 with 1 infra, its rate-limited attempt beside a resolved and a failed one; t2 at 2/3 with a timeout

The screenshots are two real standard runs of GSM8K against a stand-in model server (one problem, three attempts each); the instruction scores are stand-in records.

Verified

  • On this branch merged with main (9b7c820): python3 app/tests/parallel.py, 2198 tests, OK; uv run --no-project --with-requirements app/requirements.txt python app/tests/parallel.py app/ui_tests, 542 tests, OK.
  • New tests: test_attempts.py (numbering, by the run's own number or by start; grouping, infra apart, the verdict rule, token means, the pass@k rules), test_comparison.py (disagreement by pass rate and by endings, single attempts, infra, ranking, a pass rate never set against a mean score, pass@k over the rows' tasks), test_attempt_routes.py and test_comparison_routes.py (tabs, All attempts, grouped rows with their means and infra, pass@k on the run page, the counts in Compare and side by side).
  • Field contract refreshed: attempts.Attempts is pinned. The docs-drift check's two findings (tau3-bench/search_relay, standard/pace) are on main too.
  • Browser checks in Chromium: job page, trial page, All attempts, Compare and side by side, dark and light at 1280 px and dark at 390 px, no page errors. The task row opens by click and by Enter; a filter opens the groups it matches and clearing it restores those opened by hand; sorting keeps a group together; hovering a closed task row lights its attempts on the board; every attempt's square has a colour, cancelled and scored included.
  • A real standard run whose attempt 2 was retried after attempt 3 lists attempts 1, 2, 3 in the run's own order.
  • Two review passes on the diff. What they found is fixed here: pass@k no longer leaves out a task where the agent timed out (it read pass^1 = 100% on a run whose adjusted rate is 56%), infra attempts are apart from every count, one flipped attempt no longer outranks a steady gap, and a single attempt that ended differently is still a disagreement.

Risk

  • Compare's "N where they disagree" and its row order change for every comparison, single attempts included: a row differs by pass rate or by how its attempts ended, and ranks by the gap read with one pass and one fail added.
  • pass@k and pass^k are new numbers on the run page and in Compare; a wrong rule would mislead. docs/field-guide.md#attempts states the rules, and the tests pin them.
  • A repeated task's attempt rows are hidden until opened; if board.js fails they stay hidden, and the trial tabs and All attempts still list them.
  • The standard's pass_hat runs in write_result at the end of every standard run, so an error in it would leave that run without its result.json. It now calls the shared estimators (imported inside the function); test_pass_hat_k_as_tau_bench_counts_it pins its output, which is unchanged, and the shard tests (test_standard_shard.py) write their results through it.
  • Undo: revert the squash commit. Nothing is migrated or written.

…ith pass@k and pass^k

Trials of one task in one job are its attempts, numbered by when they
started (nullius.runs.attempts). The trial page gets a tab per attempt and
an All attempts table; the job page shows one row per task with its pass
count, the instruction score once and the mean tokens and times; the run
page and Compare show pass@k and pass^k. Compare judges a task by its pass
rate over every attempt, ranks by how far it moved (read with one pass and
one fail added, so a single flipped attempt ranks below a steady gap), and
its side-by-side page gives each run its own attempt tabs.

Attempts are read as the adjusted rate reads trials: an infra attempt is
counted apart and left out of the pass count, the means and pass@k; an
agent's timeout or crash counts as a failure. The standard's pass_hat now
calls the same estimators; what it writes is unchanged.
…nt-sorting

# Conflicts:
#	app/static/style.css
@github-actions

This comment has been minimized.

A repeat is one run of a task, counted whatever its result (Harbor's
n_attempts, the standard's --repeat). An attempt is one try within a
repeat, taken again only when the infrastructure or the agent fails, and
only its last try is kept as the repeat's trial. The pages, Compare and its
command's JSON, the field guide and the module (nullius.runs.repeats) now
say repeat where they meant one. The standard still writes pass_hat_k's
`attempts` key, the name result.json has always carried.
@github-actions

This comment has been minimized.

@aitianshi aitianshi changed the title feat(runs): read a task's repeated attempts together, by pass rate, with pass@k and pass^k feat(runs): read a task's repeats together, by pass rate, with pass@k and pass^k Nov 16, 2026
…the run does

An attempt is one run of a task, counted whatever its result (Harbor's
n_attempts, the standard's --repeat); a retry is a further try of an
attempt after an infrastructure or agent failure, and only its last try is
kept as the attempt's trial. This takes back the word repeat for the first,
as Harbor names it, and the field guide's section says both.

A task's attempts are numbered by the run's own number when every trial
carries one (the standard's ..._u0042r2), so an attempt retried after the
others keeps its place; by start time otherwise, as Harbor's names carry no
number.
@github-actions

This comment has been minimized.

aitianshi added a commit that referenced this pull request Nov 16, 2026
…the agent, with --retries beside --attempts

Merges #146 and rebuilds this change in its words. An attempt is one run
of a unit, counted whatever its result: --attempts K, the older --repeat K
still read. A retry runs an attempt again after a failure a further try
may mend, by Harbor's retry rule (a rate limit, a provider, network or
setup error, an agent crash, an output-length or verifier error; never a
timeout, a refusal, a wrong model or key, a missing secret, a usage limit,
the spending cap, a stop, or a task the agent failed): --retries N, 0 by
default, so the single re-run of a rate-limited unit now needs --retries 1.

Earlier tries are kept in retried/<trial>.try-N, the kept trial records
its retries (standard.retries) and the config records retries. The builder
says Attempts per task and Retries per attempt for both runners, and a
Harbor recipe's retries default to 0. A retried attempt says so on the run
page, the trial page, All attempts and the live board, and the progress
counts every try. The driver protocol, the catalogue and the recipe formats
keep main's words.
@aitianshi aitianshi changed the title feat(runs): read a task's repeats together, by pass rate, with pass@k and pass^k feat(runs): read a task's repeated attempts together, by pass rate, with pass@k and pass^k Nov 16, 2026
…hanges a reading

Keep main's pass@k and pass^k text; add only the words for attempt and retry, which tasks the pages leave out, how a task's pass rate counts and when a Compare row disagrees.
aitianshi added a commit that referenced this pull request Nov 17, 2026
…empts

Brings in main through #146. The conflicts are resolved as in the rebased branch the suites passed on; the resolution also cuts the retries docs and the --retries help to the AGENTS.md documentation rule.
@github-actions

Copy link
Copy Markdown

🟡 PR judge · 6 of 8 yes · #146 ↔ #130

Merge risk 🔴 MAJOR the judge's reading of the diff; app/nullius/standard/job.py is in the path of every run
Priority 🔴 EXTREME the deepest path is app/nullius/standard/job.py
Question Verdict
Changes do what the issue asks · 8 of 8 criteria ✅ yes
Issue and pull request describe the same change ⚠️ partly
Description matches the diff, no more, no less ✅ yes
Diff stays in scope ✅ yes
Tests cover the change, or Verified says how ✅ yes
Docs follow the change ✅ yes
Risk named in the description ⚠️ partly
Before and after shown · 11 images · the diff touches what is visible ✅ yes
Acceptance criteria, 8 of 8 met
# Criterion Verdict
1 A task row carries, per run, every attempt in the cohort, read as the adjusted rate reads trials: the passes over the a… ✅ met (cut)
2 differs and n_differing compare those per-run rates and how the attempts that ran ended: a task with the same pass … ✅ met (cut)
3 Rows sort by how far the pass rate moved (for a scored task, the mean score), each rate read with one pass and one fail… ✅ met (cut)
4 Each cell shows the pass count as passes/attempts when a run has more than one attempt, infra attempts apart; the lin… ✅ met (cut)
5 A test builds two runs with repeated attempts and checks: 2/3 against 2/3 is not a disagreement, 0/3 against 3/3 sorts … ✅ met (cut)
6 When every run has two or more attempts per task, the run summaries show pass@k and pass^k for k from 1 to the smallest… ✅ met (cut)
7 One implementation of the two estimators over (n, c) per task, used both by this view and by pass_hat in `app/nullius… ✅ met (cut)
8 docs/field-guide.md and the compare page's wording describe disagreement by pass rate; the field contract is refreshe… ✅ met (cut)
Checks the script runs itself, 4 of 4 pass
No machine attribution trailer in the commits or the description ✓
New code under app/, docs/, .github/ or .claude/ ✓
No Flask import under the backend package ✓
No key-shaped string in the diff ✓

some of the diff was cut before the judge read it

@aitianshi
aitianshi merged commit fe6f0a7 into main Nov 17, 2026
2 checks passed
@aitianshi
aitianshi deleted the feat/compare-pass-count-sorting branch November 17, 2026 22:19
aitianshi added a commit that referenced this pull request Nov 17, 2026
…the agent, with --retries beside --attempts (#154)

Closes #152

## What

`--retries N` (0 by default, at most 9) runs a failed attempt again
after the other units, within `-j`; its last try counts. `--attempts K`
is the new name of `--repeat K`, which still works.

- Retried: a rate limit, after the provider's pause; a provider, network
or setup error; an agent crash or setup timeout; a run or check out of
time while the provider refused or failed the calls.
- Not retried: a wrong key or spent credit (three in a row stop the
run), a setup this machine lacks, a missing secret, a spending or call
cap, a Stop, a failed task. A unit's cap counts all its tries.
- Before a run: a model missing from the provider's model list is
refused. A capped run fetches the price list first; offline, with a
model it cannot price, the run goes on uncapped and says so.
- Earlier tries are kept under `retried/<trial>.try-N/`: Try tabs on the
attempt's trial page, a row each in All attempts, a Retries column on
the run page. A judgement stays with its try; the run's cost counts
every try.
- The builder offers Retries per attempt on both runners; a new Harbor
recipe retries nothing unless asked (3 before).

Stacked on #146.

## Touches

the standard · backend · templates and static · docs and site

## Before and after

| Before | After |
| --- | --- |
| <img width="440" alt="Standard recipe, Limits: Attempts per problem
only"
src="https://github.com/user-attachments/assets/bbd67d22-81a7-4261-96fe-bb14674ae0a4"
/> | <img width="440" alt="Standard recipe, Limits: Attempts per problem
1 and Retries per attempt 0"
src="https://github.com/user-attachments/assets/b7eb83cd-61dc-48be-890f-b895fcc5451a"
/> |
| <img width="440" alt="Harbor recipe, Limits: Retries per trial 3"
src="https://github.com/user-attachments/assets/a5f6d0aa-bb07-4954-9d6f-52eef131e1c0"
/> | <img width="440" alt="Harbor recipe, Limits: Retries per attempt 0"
src="https://github.com/user-attachments/assets/7a38f659-e576-4ae2-bca8-b40cb00c2b58"
/> |

<img width="760" alt="Live board while attempt 2 is retried: retried
once, after infra error"
src="https://github.com/user-attachments/assets/afb56f15-4d59-46f6-9d65-39658693be9f"
/>

<img width="760" alt="Harbor recipe, common settings: Attempts per task
1"
src="https://github.com/user-attachments/assets/a019db44-212d-4e6d-ad26-6902378724df"
/>

## Verified

- `python3 app/tests/parallel.py`: 2229 tests, OK. Route suite: 547
tests, OK. The tests added with each fix fail on the code before it.
- Stand-in provider: a 429 with `Retry-After: 75` waited 75 s, then
retried; a Stop during that wait or during the retry keeps the earlier
try; a model the provider refuses (400) stops the run after three
refusals, with no retry.
- OpenRouter: a 6-unit run passed 6 of 6, its bill matching the relay's
record; a made-up model name refused in 2 s, before any call; a capped
run fetched the price list and applied the cap; with HTTPS cut off, it
ran uncapped and said so.
- Site build: every link and anchor resolves. The docs-drift check's one
finding is on `main` too.

## Risk

- Nothing is retried by default: a rate-limited unit used to run once
more (#90) and now needs `--retries 1`. A new Harbor recipe retries
nothing (3 before); a saved recipe keeps its value.
- A run now asks the provider for its model list before it starts, and a
capped run fetches the price list first: a slow list delays the start;
one that cannot be read checks nothing.
- `--repeat` keeps working; recipes now write `--attempts`.
- Undo: revert the squash commit.
harenome added a commit that referenced this pull request Nov 19, 2026
Closes #217

## What

Compare gets a third view, Chart (`?view=chart`), beside Side by side
and Task by task. It has the same columns, cohort and controls row. The
column rows (header, Workspace, Run) stay above the charts, so you
change the runs as in the other views.

**Cards.** Each chart is a card. Its header is one row of toolkit
buttons, all `--tool-h` high, with Remove at its right end:

- Style, with the chart icon. Its menu lists Bars, Dots and Scatter from
the registry of styles, and a check marks the current one.
- For Bars and Dots, a Data button with the sliders icon and the chosen
metric's name. Its menu lists the side table's summary rows, in the
table's groups, and a check marks the current one.
- For Scatter, an X button and a Y button. Each has its letter in the
icon slot and its metric's name, and opens the same menu for its axis.
- Remove, the same × as a column's Remove, with the name "Remove this
chart".

Every menu item is a link, and `NulliusTable.menu` opens the menus, so
the change adds no other script. A button never breaks its name over two
lines. When a row is too narrow, whole buttons move to a second row, and
Remove stays at the end of the first. At 1280 px, a Scatter card's
Style, X and Y fit on one row, also with long metric names. A style
change keeps the card's first metric. For Scatter, Y becomes the
headline rate, or the cost when the rate is X. Add chart, in the actions
slot beside Add a run, appends a Bars card of the headline rate.

**URL.** Each card is one `chart=` value, in order:
`chart=bars:adjusted_rate&chart=scatter:cost,adjusted_rate`. The metric
keys are stable names, not labels. Every link on the page keeps them,
with the view and the cohort. Without `chart=`, the page shows three
cards: Bars of the headline rate, Bars of the cost, and a Scatter of the
cost against the rate. The headline rate is the adjusted rate, or the
mean score for a benchmark that only scores. `chart=` alone shows one
Add chart button. A card whose metric this comparison does not have
keeps its controls and says "Not in this comparison."

**Styles.**

- Bars: one bar for each run, from zero, on the metric's own axis and
unit. A dashed mark in each row shows the baseline's value.
- Dots: the same layout with dots. The axis covers the values, not zero,
so close values stay apart.
- Scatter: one dot for each run, numbered by its column, with a key
under the plot. Runs at the same point sit in a ring around it, each
with its own number. A number goes on the first free side of its dot
inside the plot.

**Values.** Each value is the side table's own cell: the number its
difference uses, and the text it shows. A missing value has no mark,
only its dash. A check is hollow and named "(check)". A check's $0.00 or
0 tokens is a real zero, as in the table, so it has a mark. Nothing is
ranked. The tick labels use the row's unit: % for rates, $ for cost, k
and M for tokens, seconds or minutes for time. A tick label that would
touch the one before it is left out, but its gridline stays.

**Colours.** Each run has one colour in every chart: its column's place
on the page picks `--series-1` to `--series-8`, and the ninth column
starts again at the first. The drawing data puts the index on each mark
as a class (`s1` to `s8`), and the stylesheet maps each class to its
token, so the template has no colour logic. A check stays hollow, with
its outline in the run's colour. The baseline mark stays a dashed grey
line. The key under a Scatter gives each run a small swatch beside its
number and name, so a colour is never the only cue.

**The series palette.** Eight new tokens, `--series-1` to `--series-8`,
in `style.css`. They are muted blues, two teals, a brown and a plum, in
the app's existing lightness range. None of them reads as an outcome
colour for a pass (sage), a failure (red) or a warning (amber). The
nearest pair is the dark brown and the warning amber, at ΔE 9.5, and
every other pair is at 12.5 or more. The trace categories keep their own
`--cat-*` colours, so a colour has one meaning on each page. The dark
values serve the default dark theme and the charcoal, graphite and
midnight palettes, and they pass on the surface of each.

| Slot | Light | Dark |
| --- | --- | --- |
| 1 | `#215da5` | `#3a75bf` |
| 2 | `#179696` | `#16a8a8` |
| 3 | `#7d3f17` | `#ad6a44` |
| 4 | `#657dd1` | `#748ee3` |
| 5 | `#7e2e59` | `#aa5d83` |
| 6 | `#1587ba` | `#3aa0d4` |
| 7 | `#5e5197` | `#7768be` |
| 8 | `#2695ac` | `#38a3ba` |

The data-viz palette validator checks the colours against the card
surface (`--surface-2`). All checks pass in both modes:

| | Light (`#edf0f4`) | Dark (`#292b2f`, `#202024`, `#191f29`,
`#18263c`) |
| --- | --- | --- |
| Lowest contrast | 3.07:1 | 3.02:1 (default dark); 3.24:1 or more on
the others |
| Adjacent slots, protan and deutan ΔE (target 8) | 10.7 or more | 10.1
or more |
| Adjacent slots, normal vision ΔE (floor 15) | 15.2 or more | 15.2 or
more |
| First 4 slots, every pair, CVD ΔE | 10.4 or more | 9.5 or more |
| First 5 slots, every pair, CVD ΔE | 9.4 or more | 8.1 or more |
| Lightness band and chroma floor | pass | pass |

ΔE is the OKLab distance ×100, with protanopia and deuteranopia
simulated by Machado-Oliveira-Fernandes at full severity. Slots 6 to 8
repeat the blue and teal families, so a scatter of more than five runs
also relies on the numbers and the key. Another view can use the palette
with the class `s1` to `s8` on a mark, or `var(--series-n)` directly,
for any set of items with a stable order.

**Chart module.** `nullius/charts.py` computes the geometry in SVG
units. `templates/_charts.html` draws the marks and texts as it gets
them, with no arithmetic. Each chart is an SVG with `role="img"` and a
title. Each mark has a tooltip, and a table for a screen reader gives
every value. A registry, `STYLES`, gives each style its label, its
drawing function and the names of the metrics it takes (`("Metric",)` or
`("X", "Y")`). The card header reads it. A new style, for example Lines
for pass@k after #146, is one function and one entry. The trace
distributions (`traces/charts.py`) put durations in bins on a log axis.
They share no scale with these charts, so they keep their own code.

Docs: the results guide names the Chart view. `AGENTS.md` lists the
module.

## Touches

backend (`nullius/charts.py`, new) · routes (`web/compare.py`) ·
templates and static (`_charts.html` new, `compare/compare.html`,
`base.html` (an `i-chart` icon), `compare/columns.js`,
`compare/comparison.css`, `style.css`) · tests
(`test_categorical_charts.py` new, `test_comparison_routes.py`,
`field_contract.json`) · docs (`docs/results.md`, `AGENTS.md`)

## Before and after

Seeded data: a showcase profile of Terminal-Bench 2.0 runs, seven
compared (six agents and the oracle check). The view is new, so there is
no before. The nine-run shots add the `nop` check and a copy of the
first run.

| | After |
| --- | --- |
| The default cards, 1280 px, light | <img width="420"
alt="default-light-1280"
src="https://github.com/user-attachments/assets/8c5a551f-8969-4732-ac59-de5b2abd98be"/>
|
| The default cards, 390 px, dark | <img width="420"
alt="default-dark-390"
src="https://github.com/user-attachments/assets/7f4f9889-8ecd-440e-8e69-121b242e477e"/>
|
| Dots and a scatter, 1280 px, light | <img width="420"
alt="styles-light-1280"
src="https://github.com/user-attachments/assets/91ac0bbb-3990-44df-80c9-2348e9f56531"/>
|
| Dots and a scatter, 390 px, dark | <img width="420"
alt="styles-dark-390"
src="https://github.com/user-attachments/assets/f91782ed-b1d5-4309-9e26-e0c8cb1120a8"/>
|
| Two scatters with long metric names, 1280 px, light | <img width="420"
alt="long-names-light-1280"
src="https://github.com/user-attachments/assets/6db5d8cf-5977-4b29-9c49-02d613d10edf"/>
|
| Two scatters with long metric names, 390 px, dark | <img width="420"
alt="long-names-dark-390"
src="https://github.com/user-attachments/assets/4b55313a-3667-45f3-b4b1-1183fe40cad9"/>
|
| Seven runs at one point, 1280 px, light | <img width="420"
alt="ties-light-1280"
src="https://github.com/user-attachments/assets/5e8c8d60-b1a0-49f0-8f9a-65193d8fb294"/>
|
| Seven runs at one point, 390 px, dark | <img width="420"
alt="ties-dark-390"
src="https://github.com/user-attachments/assets/7600b4bf-3363-4a48-88b7-733a78f2b4ba"/>
|
| A card's Style menu, 1280 px, light | <img width="420"
alt="style-menu-light-1280"
src="https://github.com/user-attachments/assets/26d2e36f-be2c-4657-be4c-0281164fddfb"/>
|
| After Add chart, Remove and a style change, 1280 px, light | <img
width="420" alt="edited-light-1280"
src="https://github.com/user-attachments/assets/489bc6a5-dd7c-449e-9977-c4c306010194"/>
|
| After Add chart, Remove and a style change, 390 px, dark | <img
width="420" alt="edited-dark-390"
src="https://github.com/user-attachments/assets/e1f04391-eb6a-4d60-be19-105303cb976a"/>
|
| A Y menu opened with the keyboard, 1280 px, light | <img width="420"
alt="keyboard-menu-light-1280"
src="https://github.com/user-attachments/assets/74cd8b82-8f10-4256-8b86-ce78d4b24d03"/>
|
| A Y menu opened with the keyboard, 390 px, dark | <img width="420"
alt="keyboard-menu-dark-390"
src="https://github.com/user-attachments/assets/fa45a2be-f218-4055-90d1-e18577383289"/>
|
| Nine runs: eight colours, and the ninth back at the first, 1280 px,
light | <img width="420" alt="nine-light-1280"
src="https://github.com/user-attachments/assets/3be81d87-3f6e-4032-857d-29e24a80be06"/>
|
| Nine runs, 390 px, dark | <img width="420" alt="nine-dark-390"
src="https://github.com/user-attachments/assets/979167fb-990a-4d07-8450-b30fcd1241bc"/>
|
| No chart, 1280 px, light | <img width="420" alt="empty-light-1280"
src="https://github.com/user-attachments/assets/17d767e2-d83b-415a-abca-4789ed4de403"/>
|
| The trial page's distributions, unchanged, 1280 px, light | <img
width="420" alt="trial-distributions-light-1280"
src="https://github.com/user-attachments/assets/78c020b2-97cd-494c-b6dd-4ea338180e07"/>
|

## Verified

In a throwaway rootless `python:3.12` container with Node.js and uv, on
a copy of the branch:

- `python3 app/tests/parallel.py`: 3004 tests in 154 modules, OK.
- `pip install -r app/requirements.txt`, then `python3
app/tests/parallel.py app/ui_tests`: 655 tests in 54 modules, OK.
- New: `test_categorical_charts.py` covers the ticks and tick labels
that would touch, a gap for a missing value, a zero and a negative
value, a check, the baseline mark, a single run, a long name that wraps,
very different magnitudes, a scatter of runs at one point, the numbers
inside the plot, the registry, and each mark's series class, which
cycles after eight. `test_comparison_routes.py` covers the default
cards, the URL shape, Add chart, Remove, a style change that keeps the
metric, the Data, X and Y menus and the names on their buttons, no
chart, a metric the cohort does not have, a check run, one run and an
empty column, the column colours of eleven runs in every style, and the
`--series-*` tokens in the dark and the light theme.

Browser check, Chromium (Playwright 1.49.1), on the showcase data with
seven runs, at 1280 px light and 390 px dark:

- Every non-check mark of a card has its run's colour: six fills for the
six agents, with the oracle and `nop` hollow. With nine runs, the ninth
has the first run's colour.
- Every Scatter card shows a dot and a number for each run with both
values, and no number is outside its plot. That includes Trials against
Broke tests, where the seven runs share one point.
- Each card's header buttons are all 34 px high. At 1280 px every header
is one row, the scatters with long metric names too. At 390 px a Scatter
header takes two rows, with whole buttons on each.
- Add chart gives four cards. Remove on the second card gives three, and
the others stay in order. Dots in the first card's Style menu changes
only that card.
- Keyboard: Enter on the Y button opens its menu with the focus on the
first metric, and Tab and Enter pick the next one. An open menu does not
widen the page.
- No page JavaScript error. No sideways scroll on the Compare page.
- A trial page with Distributions: its HTML on `main` and on this branch
is the same, except the form tokens, the interface hash and the new icon
in the sprite.

Not checked: other browsers, a screen reader, touch.

## Risk

- The view toggle has a third link, and `?view=chart` is new. Every
other value of `view` still opens Side by side.
- `summary_rows` now gives each row a key, a group, its number and its
unit. The side table reads none of these, so its cells do not change.
- Every menu item is a full link, so a page with many cards carries many
long URLs (one for each metric of each axis).
- A menu choice reloads the page, so the focus goes back to the top.
- A value's text is the table's text. Where the table shows a dash for a
real 0 (Infra errors, Broke tests), the chart draws a zero mark beside
the dash.
- To undo, revert the two commits. No stored data changes.

## Related

- #206 and #207, the Compare redesign.
- #146, repeated attempts and pass@k, for a later Lines style.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant