Repository navigation
feat(runs): read a task's repeated attempts together, by pass rate, with pass@k and pass^k - #146
Merged
Merged
Conversation
…ith pass@k and pass^k Trials of one task in one job are its attempts, numbered by when they started (nullius.runs.attempts). The trial page gets a tab per attempt and an All attempts table; the job page shows one row per task with its pass count, the instruction score once and the mean tokens and times; the run page and Compare show pass@k and pass^k. Compare judges a task by its pass rate over every attempt, ranks by how far it moved (read with one pass and one fail added, so a single flipped attempt ranks below a steady gap), and its side-by-side page gives each run its own attempt tabs. Attempts are read as the adjusted rate reads trials: an infra attempt is counted apart and left out of the pass count, the means and pass@k; an agent's timeout or crash counts as a failure. The standard's pass_hat now calls the same estimators; what it writes is unchanged.
…nt-sorting # Conflicts: # app/static/style.css
Closed
3 of 15 tasks
This comment has been minimized.
This comment has been minimized.
A repeat is one run of a task, counted whatever its result (Harbor's n_attempts, the standard's --repeat). An attempt is one try within a repeat, taken again only when the infrastructure or the agent fails, and only its last try is kept as the repeat's trial. The pages, Compare and its command's JSON, the field guide and the module (nullius.runs.repeats) now say repeat where they meant one. The standard still writes pass_hat_k's `attempts` key, the name result.json has always carried.
This comment has been minimized.
This comment has been minimized.
…the run does An attempt is one run of a task, counted whatever its result (Harbor's n_attempts, the standard's --repeat); a retry is a further try of an attempt after an infrastructure or agent failure, and only its last try is kept as the attempt's trial. This takes back the word repeat for the first, as Harbor names it, and the field guide's section says both. A task's attempts are numbered by the run's own number when every trial carries one (the standard's ..._u0042r2), so an attempt retried after the others keeps its place; by start time otherwise, as Harbor's names carry no number.
This comment has been minimized.
This comment has been minimized.
aitianshi
added a commit
that referenced
this pull request
Nov 16, 2026
…the agent, with --retries beside --attempts Merges #146 and rebuilds this change in its words. An attempt is one run of a unit, counted whatever its result: --attempts K, the older --repeat K still read. A retry runs an attempt again after a failure a further try may mend, by Harbor's retry rule (a rate limit, a provider, network or setup error, an agent crash, an output-length or verifier error; never a timeout, a refusal, a wrong model or key, a missing secret, a usage limit, the spending cap, a stop, or a task the agent failed): --retries N, 0 by default, so the single re-run of a rate-limited unit now needs --retries 1. Earlier tries are kept in retried/<trial>.try-N, the kept trial records its retries (standard.retries) and the config records retries. The builder says Attempts per task and Retries per attempt for both runners, and a Harbor recipe's retries default to 0. A retried attempt says so on the run page, the trial page, All attempts and the live board, and the progress counts every try. The driver protocol, the catalogue and the recipe formats keep main's words.
This was referenced Nov 16, 2026
…hanges a reading Keep main's pass@k and pass^k text; add only the words for attempt and retry, which tasks the pages leave out, how a task's pass rate counts and when a Compare row disagrees.
aitianshi
added a commit
that referenced
this pull request
Nov 17, 2026
…empts Brings in main through #146. The conflicts are resolved as in the rebased branch the suites passed on; the resolution also cuts the retries docs and the --retries help to the AGENTS.md documentation rule.
🟡 PR judge · 6 of 8 yes · #146 ↔ #130
Acceptance criteria, 8 of 8 met
Checks the script runs itself, 4 of 4 pass
some of the diff was cut before the judge read it |
aitianshi
added a commit
that referenced
this pull request
Nov 17, 2026
…the agent, with --retries beside --attempts (#154) Closes #152 ## What `--retries N` (0 by default, at most 9) runs a failed attempt again after the other units, within `-j`; its last try counts. `--attempts K` is the new name of `--repeat K`, which still works. - Retried: a rate limit, after the provider's pause; a provider, network or setup error; an agent crash or setup timeout; a run or check out of time while the provider refused or failed the calls. - Not retried: a wrong key or spent credit (three in a row stop the run), a setup this machine lacks, a missing secret, a spending or call cap, a Stop, a failed task. A unit's cap counts all its tries. - Before a run: a model missing from the provider's model list is refused. A capped run fetches the price list first; offline, with a model it cannot price, the run goes on uncapped and says so. - Earlier tries are kept under `retried/<trial>.try-N/`: Try tabs on the attempt's trial page, a row each in All attempts, a Retries column on the run page. A judgement stays with its try; the run's cost counts every try. - The builder offers Retries per attempt on both runners; a new Harbor recipe retries nothing unless asked (3 before). Stacked on #146. ## Touches the standard · backend · templates and static · docs and site ## Before and after | Before | After | | --- | --- | | <img width="440" alt="Standard recipe, Limits: Attempts per problem only" src="https://github.com/user-attachments/assets/bbd67d22-81a7-4261-96fe-bb14674ae0a4" /> | <img width="440" alt="Standard recipe, Limits: Attempts per problem 1 and Retries per attempt 0" src="https://github.com/user-attachments/assets/b7eb83cd-61dc-48be-890f-b895fcc5451a" /> | | <img width="440" alt="Harbor recipe, Limits: Retries per trial 3" src="https://github.com/user-attachments/assets/a5f6d0aa-bb07-4954-9d6f-52eef131e1c0" /> | <img width="440" alt="Harbor recipe, Limits: Retries per attempt 0" src="https://github.com/user-attachments/assets/7a38f659-e576-4ae2-bca8-b40cb00c2b58" /> | <img width="760" alt="Live board while attempt 2 is retried: retried once, after infra error" src="https://github.com/user-attachments/assets/afb56f15-4d59-46f6-9d65-39658693be9f" /> <img width="760" alt="Harbor recipe, common settings: Attempts per task 1" src="https://github.com/user-attachments/assets/a019db44-212d-4e6d-ad26-6902378724df" /> ## Verified - `python3 app/tests/parallel.py`: 2229 tests, OK. Route suite: 547 tests, OK. The tests added with each fix fail on the code before it. - Stand-in provider: a 429 with `Retry-After: 75` waited 75 s, then retried; a Stop during that wait or during the retry keeps the earlier try; a model the provider refuses (400) stops the run after three refusals, with no retry. - OpenRouter: a 6-unit run passed 6 of 6, its bill matching the relay's record; a made-up model name refused in 2 s, before any call; a capped run fetched the price list and applied the cap; with HTTPS cut off, it ran uncapped and said so. - Site build: every link and anchor resolves. The docs-drift check's one finding is on `main` too. ## Risk - Nothing is retried by default: a rate-limited unit used to run once more (#90) and now needs `--retries 1`. A new Harbor recipe retries nothing (3 before); a saved recipe keeps its value. - A run now asks the provider for its model list before it starts, and a capped run fetches the price list first: a slow list delays the start; one that cannot be read checks nothing. - `--repeat` keeps working; recipes now write `--attempts`. - Undo: revert the squash commit.
This was referenced Nov 18, 2026
Closed
harenome
added a commit
that referenced
this pull request
Nov 19, 2026
Closes #217 ## What Compare gets a third view, Chart (`?view=chart`), beside Side by side and Task by task. It has the same columns, cohort and controls row. The column rows (header, Workspace, Run) stay above the charts, so you change the runs as in the other views. **Cards.** Each chart is a card. Its header is one row of toolkit buttons, all `--tool-h` high, with Remove at its right end: - Style, with the chart icon. Its menu lists Bars, Dots and Scatter from the registry of styles, and a check marks the current one. - For Bars and Dots, a Data button with the sliders icon and the chosen metric's name. Its menu lists the side table's summary rows, in the table's groups, and a check marks the current one. - For Scatter, an X button and a Y button. Each has its letter in the icon slot and its metric's name, and opens the same menu for its axis. - Remove, the same × as a column's Remove, with the name "Remove this chart". Every menu item is a link, and `NulliusTable.menu` opens the menus, so the change adds no other script. A button never breaks its name over two lines. When a row is too narrow, whole buttons move to a second row, and Remove stays at the end of the first. At 1280 px, a Scatter card's Style, X and Y fit on one row, also with long metric names. A style change keeps the card's first metric. For Scatter, Y becomes the headline rate, or the cost when the rate is X. Add chart, in the actions slot beside Add a run, appends a Bars card of the headline rate. **URL.** Each card is one `chart=` value, in order: `chart=bars:adjusted_rate&chart=scatter:cost,adjusted_rate`. The metric keys are stable names, not labels. Every link on the page keeps them, with the view and the cohort. Without `chart=`, the page shows three cards: Bars of the headline rate, Bars of the cost, and a Scatter of the cost against the rate. The headline rate is the adjusted rate, or the mean score for a benchmark that only scores. `chart=` alone shows one Add chart button. A card whose metric this comparison does not have keeps its controls and says "Not in this comparison." **Styles.** - Bars: one bar for each run, from zero, on the metric's own axis and unit. A dashed mark in each row shows the baseline's value. - Dots: the same layout with dots. The axis covers the values, not zero, so close values stay apart. - Scatter: one dot for each run, numbered by its column, with a key under the plot. Runs at the same point sit in a ring around it, each with its own number. A number goes on the first free side of its dot inside the plot. **Values.** Each value is the side table's own cell: the number its difference uses, and the text it shows. A missing value has no mark, only its dash. A check is hollow and named "(check)". A check's $0.00 or 0 tokens is a real zero, as in the table, so it has a mark. Nothing is ranked. The tick labels use the row's unit: % for rates, $ for cost, k and M for tokens, seconds or minutes for time. A tick label that would touch the one before it is left out, but its gridline stays. **Colours.** Each run has one colour in every chart: its column's place on the page picks `--series-1` to `--series-8`, and the ninth column starts again at the first. The drawing data puts the index on each mark as a class (`s1` to `s8`), and the stylesheet maps each class to its token, so the template has no colour logic. A check stays hollow, with its outline in the run's colour. The baseline mark stays a dashed grey line. The key under a Scatter gives each run a small swatch beside its number and name, so a colour is never the only cue. **The series palette.** Eight new tokens, `--series-1` to `--series-8`, in `style.css`. They are muted blues, two teals, a brown and a plum, in the app's existing lightness range. None of them reads as an outcome colour for a pass (sage), a failure (red) or a warning (amber). The nearest pair is the dark brown and the warning amber, at ΔE 9.5, and every other pair is at 12.5 or more. The trace categories keep their own `--cat-*` colours, so a colour has one meaning on each page. The dark values serve the default dark theme and the charcoal, graphite and midnight palettes, and they pass on the surface of each. | Slot | Light | Dark | | --- | --- | --- | | 1 | `#215da5` | `#3a75bf` | | 2 | `#179696` | `#16a8a8` | | 3 | `#7d3f17` | `#ad6a44` | | 4 | `#657dd1` | `#748ee3` | | 5 | `#7e2e59` | `#aa5d83` | | 6 | `#1587ba` | `#3aa0d4` | | 7 | `#5e5197` | `#7768be` | | 8 | `#2695ac` | `#38a3ba` | The data-viz palette validator checks the colours against the card surface (`--surface-2`). All checks pass in both modes: | | Light (`#edf0f4`) | Dark (`#292b2f`, `#202024`, `#191f29`, `#18263c`) | | --- | --- | --- | | Lowest contrast | 3.07:1 | 3.02:1 (default dark); 3.24:1 or more on the others | | Adjacent slots, protan and deutan ΔE (target 8) | 10.7 or more | 10.1 or more | | Adjacent slots, normal vision ΔE (floor 15) | 15.2 or more | 15.2 or more | | First 4 slots, every pair, CVD ΔE | 10.4 or more | 9.5 or more | | First 5 slots, every pair, CVD ΔE | 9.4 or more | 8.1 or more | | Lightness band and chroma floor | pass | pass | ΔE is the OKLab distance ×100, with protanopia and deuteranopia simulated by Machado-Oliveira-Fernandes at full severity. Slots 6 to 8 repeat the blue and teal families, so a scatter of more than five runs also relies on the numbers and the key. Another view can use the palette with the class `s1` to `s8` on a mark, or `var(--series-n)` directly, for any set of items with a stable order. **Chart module.** `nullius/charts.py` computes the geometry in SVG units. `templates/_charts.html` draws the marks and texts as it gets them, with no arithmetic. Each chart is an SVG with `role="img"` and a title. Each mark has a tooltip, and a table for a screen reader gives every value. A registry, `STYLES`, gives each style its label, its drawing function and the names of the metrics it takes (`("Metric",)` or `("X", "Y")`). The card header reads it. A new style, for example Lines for pass@k after #146, is one function and one entry. The trace distributions (`traces/charts.py`) put durations in bins on a log axis. They share no scale with these charts, so they keep their own code. Docs: the results guide names the Chart view. `AGENTS.md` lists the module. ## Touches backend (`nullius/charts.py`, new) · routes (`web/compare.py`) · templates and static (`_charts.html` new, `compare/compare.html`, `base.html` (an `i-chart` icon), `compare/columns.js`, `compare/comparison.css`, `style.css`) · tests (`test_categorical_charts.py` new, `test_comparison_routes.py`, `field_contract.json`) · docs (`docs/results.md`, `AGENTS.md`) ## Before and after Seeded data: a showcase profile of Terminal-Bench 2.0 runs, seven compared (six agents and the oracle check). The view is new, so there is no before. The nine-run shots add the `nop` check and a copy of the first run. | | After | | --- | --- | | The default cards, 1280 px, light | <img width="420" alt="default-light-1280" src="https://github.com/user-attachments/assets/8c5a551f-8969-4732-ac59-de5b2abd98be"/> | | The default cards, 390 px, dark | <img width="420" alt="default-dark-390" src="https://github.com/user-attachments/assets/7f4f9889-8ecd-440e-8e69-121b242e477e"/> | | Dots and a scatter, 1280 px, light | <img width="420" alt="styles-light-1280" src="https://github.com/user-attachments/assets/91ac0bbb-3990-44df-80c9-2348e9f56531"/> | | Dots and a scatter, 390 px, dark | <img width="420" alt="styles-dark-390" src="https://github.com/user-attachments/assets/f91782ed-b1d5-4309-9e26-e0c8cb1120a8"/> | | Two scatters with long metric names, 1280 px, light | <img width="420" alt="long-names-light-1280" src="https://github.com/user-attachments/assets/6db5d8cf-5977-4b29-9c49-02d613d10edf"/> | | Two scatters with long metric names, 390 px, dark | <img width="420" alt="long-names-dark-390" src="https://github.com/user-attachments/assets/4b55313a-3667-45f3-b4b1-1183fe40cad9"/> | | Seven runs at one point, 1280 px, light | <img width="420" alt="ties-light-1280" src="https://github.com/user-attachments/assets/5e8c8d60-b1a0-49f0-8f9a-65193d8fb294"/> | | Seven runs at one point, 390 px, dark | <img width="420" alt="ties-dark-390" src="https://github.com/user-attachments/assets/7600b4bf-3363-4a48-88b7-733a78f2b4ba"/> | | A card's Style menu, 1280 px, light | <img width="420" alt="style-menu-light-1280" src="https://github.com/user-attachments/assets/26d2e36f-be2c-4657-be4c-0281164fddfb"/> | | After Add chart, Remove and a style change, 1280 px, light | <img width="420" alt="edited-light-1280" src="https://github.com/user-attachments/assets/489bc6a5-dd7c-449e-9977-c4c306010194"/> | | After Add chart, Remove and a style change, 390 px, dark | <img width="420" alt="edited-dark-390" src="https://github.com/user-attachments/assets/e1f04391-eb6a-4d60-be19-105303cb976a"/> | | A Y menu opened with the keyboard, 1280 px, light | <img width="420" alt="keyboard-menu-light-1280" src="https://github.com/user-attachments/assets/74cd8b82-8f10-4256-8b86-ce78d4b24d03"/> | | A Y menu opened with the keyboard, 390 px, dark | <img width="420" alt="keyboard-menu-dark-390" src="https://github.com/user-attachments/assets/fa45a2be-f218-4055-90d1-e18577383289"/> | | Nine runs: eight colours, and the ninth back at the first, 1280 px, light | <img width="420" alt="nine-light-1280" src="https://github.com/user-attachments/assets/3be81d87-3f6e-4032-857d-29e24a80be06"/> | | Nine runs, 390 px, dark | <img width="420" alt="nine-dark-390" src="https://github.com/user-attachments/assets/979167fb-990a-4d07-8450-b30fcd1241bc"/> | | No chart, 1280 px, light | <img width="420" alt="empty-light-1280" src="https://github.com/user-attachments/assets/17d767e2-d83b-415a-abca-4789ed4de403"/> | | The trial page's distributions, unchanged, 1280 px, light | <img width="420" alt="trial-distributions-light-1280" src="https://github.com/user-attachments/assets/78c020b2-97cd-494c-b6dd-4ea338180e07"/> | ## Verified In a throwaway rootless `python:3.12` container with Node.js and uv, on a copy of the branch: - `python3 app/tests/parallel.py`: 3004 tests in 154 modules, OK. - `pip install -r app/requirements.txt`, then `python3 app/tests/parallel.py app/ui_tests`: 655 tests in 54 modules, OK. - New: `test_categorical_charts.py` covers the ticks and tick labels that would touch, a gap for a missing value, a zero and a negative value, a check, the baseline mark, a single run, a long name that wraps, very different magnitudes, a scatter of runs at one point, the numbers inside the plot, the registry, and each mark's series class, which cycles after eight. `test_comparison_routes.py` covers the default cards, the URL shape, Add chart, Remove, a style change that keeps the metric, the Data, X and Y menus and the names on their buttons, no chart, a metric the cohort does not have, a check run, one run and an empty column, the column colours of eleven runs in every style, and the `--series-*` tokens in the dark and the light theme. Browser check, Chromium (Playwright 1.49.1), on the showcase data with seven runs, at 1280 px light and 390 px dark: - Every non-check mark of a card has its run's colour: six fills for the six agents, with the oracle and `nop` hollow. With nine runs, the ninth has the first run's colour. - Every Scatter card shows a dot and a number for each run with both values, and no number is outside its plot. That includes Trials against Broke tests, where the seven runs share one point. - Each card's header buttons are all 34 px high. At 1280 px every header is one row, the scatters with long metric names too. At 390 px a Scatter header takes two rows, with whole buttons on each. - Add chart gives four cards. Remove on the second card gives three, and the others stay in order. Dots in the first card's Style menu changes only that card. - Keyboard: Enter on the Y button opens its menu with the focus on the first metric, and Tab and Enter pick the next one. An open menu does not widen the page. - No page JavaScript error. No sideways scroll on the Compare page. - A trial page with Distributions: its HTML on `main` and on this branch is the same, except the form tokens, the interface hash and the new icon in the sprite. Not checked: other browsers, a screen reader, touch. ## Risk - The view toggle has a third link, and `?view=chart` is new. Every other value of `view` still opens Side by side. - `summary_rows` now gives each row a key, a group, its number and its unit. The side table reads none of these, so its cells do not change. - Every menu item is a full link, so a page with many cards carries many long URLs (one for each metric of each axis). - A menu choice reloads the page, so the focus goes back to the top. - A value's text is the table's text. Where the table shows a dash for a real 0 (Infra errors, Broke tests), the chart draws a zero mark beside the dash. - To undo, revert the two commits. No stored data changes. ## Related - #206 and #207, the Compare redesign. - #146, repeated attempts and pass@k, for a later Lines style.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #130
Closes #145
What
An attempt is one run of a task, counted whatever its result (Harbor's
n_attempts, the standard's--repeat); a retry is a further try of an attempt after an infrastructure or agent failure, and only its last try is kept as the attempt's trial. A task's attempts in one job are now read together, by one backend module (nullius.runs.attempts), the way the run's adjusted rate reads trials: an infra attempt is counted apart and left out of the pass count, the means and pass@k, and an agent's timeout or crash counts as a failure. Attempts are numbered by the run's own number when its trials carry one (the standard'su0000r2), so an attempt retried after the others keeps its place, else by start time. The trial page gets a tab per attempt and an All attempts table. The job page shows one row per repeated task (its pass count, a square per attempt, the instruction score once, the mean tokens and times) that opens onto its attempts, and the run page shows pass@k and pass^k. Compare (#130) judges a task by its pass rate over every attempt instead of its last one, ranks the rows by how far that rate moved (read with one pass and one fail added, so a single flipped attempt ranks below a steady gap), shows pass@k and pass^k in its run summary, and gives each side of the side-by-side page its own attempt tabs. The standard'spass_hatcalls the same estimators; what it writes is unchanged.Touches
backend · routes · templates and static · docs and site · the standard (
pass_hatcalls the shared estimators)Before and after
An infra attempt counted apart, after (a hand-made run in Harbor's layout): t1 reads 1/2 · 1 infra and its means leave the rate-limited attempt out; t2's timeout counts as a failure.
The screenshots are two real standard runs of GSM8K against a stand-in model server (one problem, three attempts each); the instruction scores are stand-in records.
Verified
main(9b7c820):python3 app/tests/parallel.py, 2198 tests, OK;uv run --no-project --with-requirements app/requirements.txt python app/tests/parallel.py app/ui_tests, 542 tests, OK.test_attempts.py(numbering, by the run's own number or by start; grouping, infra apart, the verdict rule, token means, the pass@k rules),test_comparison.py(disagreement by pass rate and by endings, single attempts, infra, ranking, a pass rate never set against a mean score, pass@k over the rows' tasks),test_attempt_routes.pyandtest_comparison_routes.py(tabs, All attempts, grouped rows with their means and infra, pass@k on the run page, the counts in Compare and side by side).attempts.Attemptsis pinned. The docs-drift check's two findings (tau3-bench/search_relay,standard/pace) are onmaintoo.Risk
docs/field-guide.md#attemptsstates the rules, and the tests pin them.board.jsfails they stay hidden, and the trial tabs and All attempts still list them.pass_hatruns inwrite_resultat the end of every standard run, so an error in it would leave that run without itsresult.json. It now calls the shared estimators (imported inside the function);test_pass_hat_k_as_tau_bench_counts_itpins its output, which is unchanged, and the shard tests (test_standard_shard.py) write their results through it.