Repository navigation
feat(standard): retry an attempt that fails on the infrastructure or the agent, with --retries beside --attempts - #154
Merged
Merged
Conversation
…ith pass@k and pass^k Trials of one task in one job are its attempts, numbered by when they started (nullius.runs.attempts). The trial page gets a tab per attempt and an All attempts table; the job page shows one row per task with its pass count, the instruction score once and the mean tokens and times; the run page and Compare show pass@k and pass^k. Compare judges a task by its pass rate over every attempt, ranks by how far it moved (read with one pass and one fail added, so a single flipped attempt ranks below a steady gap), and its side-by-side page gives each run its own attempt tabs. Attempts are read as the adjusted rate reads trials: an infra attempt is counted apart and left out of the pass count, the means and pass@k; an agent's timeout or crash counts as a failure. The standard's pass_hat now calls the same estimators; what it writes is unchanged.
…nt-sorting # Conflicts: # app/static/style.css
A repeat is one run of a task, counted whatever its result (Harbor's n_attempts, the standard's --repeat). An attempt is one try within a repeat, taken again only when the infrastructure or the agent fails, and only its last try is kept as the repeat's trial. The pages, Compare and its command's JSON, the field guide and the module (nullius.runs.repeats) now say repeat where they meant one. The standard still writes pass_hat_k's `attempts` key, the name result.json has always carried.
…tructure or the agent fails A repeat is one run of a unit, counted whatever its result (--repeat); an attempt is one try within a repeat. A repeat that ends in a failure a further try may mend (Harbor's retry rule: a rate limit, a provider, network or setup error, an agent crash, an output-length or verifier error) runs again after the other units, one at a time and paced, until it ends otherwise or has had --attempts tries, 2 by default: the single re-run of a rate-limited unit, extended to every retryable failure. Never a timeout, a refusal, a wrong model or key, a missing secret, a usage limit, the spending cap, a stop, or a task the agent failed. Each earlier try is kept in retried/<trial>.attempt-N; the kept trial records its attempt and the config records attempts. Both runners' recipes carry repeat and attempts: a Harbor recipe is saved as version 2 (repeat is --n-attempts, attempts is --max-retries + 1) and version 1 is read as it was saved. Where "attempt" named a repeat it now says repeat: the builder's labels, budgets, benchmark notes, and the driver protocol, which sends repeat to a definition declaring limits.repeats and attempt to one still declaring limits.attempts.
This comment has been minimized.
This comment has been minimized.
…rd's retry mark for both runners
…the run does An attempt is one run of a task, counted whatever its result (Harbor's n_attempts, the standard's --repeat); a retry is a further try of an attempt after an infrastructure or agent failure, and only its last try is kept as the attempt's trial. This takes back the word repeat for the first, as Harbor names it, and the field guide's section says both. A task's attempts are numbered by the run's own number when every trial carries one (the standard's ..._u0042r2), so an attempt retried after the others keeps its place; by start time otherwise, as Harbor's names carry no number.
…the agent, with --retries beside --attempts Merges #146 and rebuilds this change in its words. An attempt is one run of a unit, counted whatever its result: --attempts K, the older --repeat K still read. A retry runs an attempt again after a failure a further try may mend, by Harbor's retry rule (a rate limit, a provider, network or setup error, an agent crash, an output-length or verifier error; never a timeout, a refusal, a wrong model or key, a missing secret, a usage limit, the spending cap, a stop, or a task the agent failed): --retries N, 0 by default, so the single re-run of a rate-limited unit now needs --retries 1. Earlier tries are kept in retried/<trial>.try-N, the kept trial records its retries (standard.retries) and the config records retries. The builder says Attempts per task and Retries per attempt for both runners, and a Harbor recipe's retries default to 0. A retried attempt says so on the run page, the trial page, All attempts and the live board, and the progress counts every try. The driver protocol, the catalogue and the recipe formats keep main's words.
This comment has been minimized.
This comment has been minimized.
…at a further try may mend A retried attempt's tries are tabs of their own on its trial page (Try 1, Try 2, the last counted), and a faded row each in All attempts; the run page and All attempts count each attempt's retries against the run's limit. A retry follows only a failure a second try may mend: never a wrong key or model, spent credit, a time run out, a setup the machine cannot give, or an attempt that spent its own cap (the cap counts all of its tries together). A rate limit's retry waits the provider's pause, or a minute. A Stop during a retry keeps the try before; a merge keeps the shards' tries; a host run's copy, a judgement, the progress and the live board each read the retry as its own try. Proof runs retry a rate limit once.
…hanges a reading Keep main's pass@k and pass^k text; add only the words for attempt and retry, which tasks the pages leave out, how a task's pass rate counts and when a Compare row disagrees.
…empts Brings in main through #146. The conflicts are resolved as in the rebased branch the suites passed on; the resolution also cuts the retries docs and the --retries help to the AGENTS.md documentation rule.
… and say the pause in whole seconds A rate-limited attempt stays on the run page while its retry waits, and a Stop during the wait leaves nothing to put back; a stopped retry's empty retried/ folder goes.
…ks and attempts apart in its last line
A retry round shares the run's concurrency (-j) instead of running one at a
time; a retry stopped under it still leaves the try before. The run's last
line says the tasks and, when each was attempted more than once, the
attempts beside them ("3 problems, 6 attempts"), never the attempts as tasks.
Each earlier try was paid for: the job's total cost adds what each attempt's tries before its last recorded (Trial.retried_cost_usd), and the live board's total keeps an attempt's cost when its retry starts.
A try's page found its try in the attempt's tries, then read them all again for its tabs.
…m's checks included The catalogue's notes, conform's flag and its pages, the import-agent skill and the run's help name the flag --attempts; --repeat is still accepted.
…st, and fetch the price list a cap needs A model the provider's list lacks would be refused on every call: the run is refused before anything installs (a :suffix routes the model it follows; a provider with no list is not checked). A capped run fetches the model price list first; when it cannot be fetched and a model has no price, the run goes on without the cap and says so, in its log and on its units. No test reaches past this machine: tests/__init__.py refuses any other address.
…ck among the run's refusals The standard's Attempts and Retries rows link the field guide and the run page instead of restating them; the retries table keeps which failures are tried again and drops how many run at once and where earlier tries are kept.
This comment has been minimized.
This comment has been minimized.
🟡 PR judge · 7 of 8 yes · #154 ↔ #152
Acceptance criteria, 8 of 8 met
Checks the script runs itself, 4 of 4 pass
some of the diff was cut before the judge read it |
aitianshi
pushed a commit
that referenced
this pull request
Nov 19, 2026
…viders, set its own limits and prove a merged null control (#133) ## What A benchmark of the standard can now say more about how its units run and end, which a computer-use benchmark like OSWorld2 (#134) needs. `gated`, the new limits and `prove`'s new options are opt-in. The relay's rules and the rules of how a unit ends change for every benchmark of the standard; Risk lists them. - **Gated tasks** (`gated: true`): a unit's failure that the relay's records explain, a model endpoint's (the agent's, the judge's or the simulated user's), says in its `result.json` the status and class alone, never the provider's words, which can quote the request. The trial's call records keep the provider's words, and an error a driver raises itself is kept as the driver words it. - **A benchmark option's values by name** (`names`: {value: name}): the builder's cards and select, Review and the benchmark's page show the name, `about` stays its tooltip, and a recipe and the CLI keep the value. OSWorld1's agents are named as their upstream spells them. - **An option on its own step** (`step`): `where`, where the work runs, on the Where step, one value as fixed, and in Review's Where row; `dataset`, what some datasets need, on the Dataset step for those datasets. OSWorld1's VM provider is on Where. - **Limits of a benchmark's own:** - `limits.score_seconds`: how long `score` may take, the benchmark's level below `--timeout score=` (#191); the Limits step shows it as the benchmark's; - `limits.call_seconds`: how long the relay waits for one model call, also `relay serve --timeout`; - **How a unit ends:** - a driver's error that begins `EnvironmentSetupError: `, `VerifierError: `, `CancelledError: ` or `CauseUnknownError: ` keeps that kind; - the agent's own last call refused as longer than the model's context window is `ContextWindowExceededError`, the agent's; - a last 429 is read from the agent's own calls in the order they began, so a side model's later 200 does not cover it; - a unit a guard's stop ended keeps the score its driver gave, its own last call's 429 notwithstanding, unless that 429 said the credit is spent. `score` can say the stop's reason (`verifier_result.stopped`); - with #154's `--retries`, a unit a guard asked to stop and a full context window are not tried again; - for an upstream's own answerer: `OutputLengthExceededError` only when its last reply has neither text nor a tool call. When the answerer fails and its last reply was a Responses object answered 200 and `failed`, that is an API error. - **The relay:** - a non-streamed 200 to a model call with no reply in it reaches the caller and the record as a 502, or the 429 or 402 of an error it held (the record keeps `answered: 200`). #124 leaves counting an error inside a 200 to a pull request of its own; this one takes only a 200 with no reply in it; - a stream cut off midway, after the caller had its 200, is recorded as a 502, and its booking against the spending cap stays whole; - a call refused with 400, 413 or 422 records each inline image it sent by its declared type, the type its bytes show and its size, never a byte; - a `stop` file beside the relay's records: the relay notes in it an agent's call refused for the model's context window, and that note refuses nothing. Once a benchmark's guard writes a line of its own there, the agent's later calls are refused unsent, while side models and the judge's and simulated user's routes go on; - a failed Responses object with no usage records a cost of 0, and its booking is kept. - **A job's name:** - a run holds its job name until its process ends, and one under a name another run still holds is refused before it writes anything; - a rerun into the same job moves each rerun trial's earlier records (`.work/relay`, `result.json`, `verifier/`, `agent/`) aside to `.work/earlier.<ns>`. - **prove and merge:** - `nullius standard prove ... --bench-option NAME=VALUE` sets the benchmark's own options for the units check and the controls; - `--job nop=MERGED` (or `oracle=`) checks a merged launch of a control against its bound instead of running it, and every trial's own record; - `merge` accepts shards that differ in benchmark options and keeps each shard's in `merged_from`; - `merge` refuses, unless `--force`, a trial whose record names another job, benchmark, version, control or agent, and a job itself merged despite differences. It lists the shards absent from a merge only when the launch has at most 1,000. - **Smaller:** - a trace that cannot be built leaves `agent/trace.error.txt`; - an Original's trajectory keeps its tool calls; - each call's record keeps the request's sha256; - `thinking_tokens` count as reasoning; - a setup's `comparable` can list the models whose runs compare with the paper, and the builder's message for another model says so; - a test checks that every file a host's snapshot takes of a bundled entry is package data. **Read first:** `failure`, then `_hollow` and the stop in `Relay._forward` (`app/nullius/standard/relay.py`). Next, the outcome rules in `Run` (`app/nullius/standard/run.py`), then `names` and `step` in `recipe.compile` and `benchOptions` in the builder, the `--job` check in `prove.py` and the trial check in `merge.py`. Then `test_standard_gated.py`, `test_standard_context_window.py` and `test_standard_controls.py`. The rest is the definitions' checks, the CLI options, the sidecar's argument and the pages. ## Touches the standard (`app/nullius/standard/`: `relay`, `run`, `prove`, `definitions`, `merge`, `job`, `cli`, `container`, `recipe`, `fidelity`, `retries`) · the builder (`app/static/builder/`, `app/web/builder.py`: the score's default on the Limits step, an option's names, and an option on its own step) · the benchmark page (an option's names) · OSWorld1's definition (its agents' names, its VM provider on Where) · docs and site (`docs/standard.md`, `docs/cli/run.md`, `docs/cli/read-jobs.md`, `docs/builder.md`, `docs/settings/model-providers.md`) ## Verified - At `e768c632`, on main at `db6e8f96`, on Python 3.12: backend suite (`python3 app/tests/parallel.py`), 2310 tests in 129 modules, OK; route suite with Node on PATH, 639 tests in 53 modules, OK; the wheel's check (`app/packaging/verify.py`), OK. - **An option's names and its step, in a headless Chromium, light and dark, 1400 and 390 px,** with #134 on top: OSWorld1's agents by name in their select, OSWorld2's in their cards; the VM provider of each as a fixed value on the Where step, shown only while a participant that takes it is chosen; OSWorld2's Website host suffix on the Dataset step for All tasks, not for Offline, no services; Served by empty on any choice of model. No page error. - **The 12 changed test modules on main's source** (`5bcfd82d`): 9 fail, 2 pass, and 1 does not end. - `test_standard_context_window`: 19 of 20; `test_standard_controls`: 17 of 43; `test_standard_relay`: 8 of 58; `test_standard_gated`: 4 of 6. - `test_standard_run`: 2 of 50; `test_standard_trace`: 2 of 20; `test_standard_parts`, `test_standard_served_by` and `test_standard_units`: 1 each. - `test_standard_shard` does not end within 4 minutes: one of its merges names 2^64 shards, and main's `merge` lists every absent one. - `test_standard_versions` and `test_dependency_surface` pass on both. - **Covered in #134, not here:** that `limits.call_seconds` reaches the relay and the sidecar's `--timeout`. OSWorld2 is the first entry to set it (`test_osworld2_muse.py`). - **#124's own tests.** Three were changed for two deliberate differences: a 200 with no reply is a 502, and a failed Responses object is read as failed. The rest of `test_standard_served_by` passes as main has it; `write_shard` in `test_standard_shard` takes #137's `reward` beside this branch's `bench_options`. - `PYTHONPATH=app python3 -m nullius standard check`: ok. `python3 app/tests/catalogue_docs.py --write`, then the check: up to date. `python3 app/tests/field_contract.py`: ok. `app/site/readme_media.py --check`: current. The docs check reports the same two findings in `AGENTS.md` as on main. - Not run: a benchmark already on main, live, on this branch; the container backend with `--timeout`; the docker tier; the site build. ## Risk - **For every benchmark of the standard:** - a non-streamed 200 with no reply is a 502, and a stream cut off midway is recorded as one; - the agent's own last call refused for its context window is the agent's error (`ContextWindowExceededError`), where it was an API error outside its rate; - an agent's last 429 followed by a side model's 200 makes the unit `rate_limited`, where its score stood; - an upstream answerer whose last reply was a `failed` Responses object is an API error, where it was the agent's; - a driver's error of a kind it names: a setup error or a stop is left out of the agent's rate, and a verifier error or one of no known cause counts as unknown. A wrong kind would move a unit between them; - `prove` runs no control when the units check failed, and fails a control job with a unit that has neither a score nor an error; - a run under a job name another run still holds is refused, and a rerun into an existing job moves the rerun trials' records aside; - `merge` refuses a trial whose record names another job unless `--force`. - **Records.** Stored records gain fields and keep the old ones. - These commits sit on main at `db6e8f96`, with #124, #150, #191, #154, #174 and the Recipes page (#205). #124's `--served-by` code is main's as merged; #191's time limits come first, so `--timeout score=` is used over `limits.score_seconds`, and a unit's cap on model calls (`UnitCallCapError`) is read before this PR's outcome rules. #154's `--retries` replaces the one more run of a rate-limited unit, so the earlier `limits.run_again: false` that switched it off is gone; main counts a provider's 402 as a usage limit, and the tests follow. The first and fourth commits gave a definition pins of its serving providers, defaults of Served by after review; the last commit removes them, as the reviewer asked, since no benchmark declares one: the diff over main has none, and Served by is the user's alone, as in #124. - The fifth commit shortens this PR's docs and messages. No logic changes. - **Names and Where.** An option's names change only what is shown; a recipe saved before keeps working. An option of where the work runs leaves the Agent step for Where, and one some datasets need is asked on Dataset: OSWorld1's VM provider is now on Where. - Undone by reverting the commits. --------- Co-authored-by: Oleg <247799477+sirotaoleg@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #152
What
--retries N(0 by default, at most 9) runs a failed attempt again after the other units, within-j; its last try counts.--attempts Kis the new name of--repeat K, which still works.retried/<trial>.try-N/: Try tabs on the attempt's trial page, a row each in All attempts, a Retries column on the run page. A judgement stays with its try; the run's cost counts every try.Stacked on #146.
Touches
the standard · backend · templates and static · docs and site
Before and after
Verified
python3 app/tests/parallel.py: 2229 tests, OK. Route suite: 547 tests, OK. The tests added with each fix fail on the code before it.Retry-After: 75waited 75 s, then retried; a Stop during that wait or during the retry keeps the earlier try; a model the provider refuses (400) stops the run after three refusals, with no retry.maintoo.Risk
--retries 1. A new Harbor recipe retries nothing (3 before); a saved recipe keeps its value.--repeatkeeps working; recipes now write--attempts.