Skip to content

feat(standard): retry an attempt that fails on the infrastructure or the agent, with --retries beside --attempts - #154

Merged
aitianshi merged 20 commits into
mainfrom
feat/standard-attempts
Nov 17, 2026
Merged

aitianshi merged 20 commits into
mainfrom
feat/standard-attempts

Conversation

@aitianshi

@aitianshi aitianshi commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #152

What

--retries N (0 by default, at most 9) runs a failed attempt again after the other units, within -j; its last try counts. --attempts K is the new name of --repeat K, which still works.

  • Retried: a rate limit, after the provider's pause; a provider, network or setup error; an agent crash or setup timeout; a run or check out of time while the provider refused or failed the calls.
  • Not retried: a wrong key or spent credit (three in a row stop the run), a setup this machine lacks, a missing secret, a spending or call cap, a Stop, a failed task. A unit's cap counts all its tries.
  • Before a run: a model missing from the provider's model list is refused. A capped run fetches the price list first; offline, with a model it cannot price, the run goes on uncapped and says so.
  • Earlier tries are kept under retried/<trial>.try-N/: Try tabs on the attempt's trial page, a row each in All attempts, a Retries column on the run page. A judgement stays with its try; the run's cost counts every try.
  • The builder offers Retries per attempt on both runners; a new Harbor recipe retries nothing unless asked (3 before).

Stacked on #146.

Touches

the standard · backend · templates and static · docs and site

Before and after

Before After
Standard recipe, Limits: Attempts per problem only Standard recipe, Limits: Attempts per problem 1 and Retries per attempt 0
Harbor recipe, Limits: Retries per trial 3 Harbor recipe, Limits: Retries per attempt 0
Live board while attempt 2 is retried: retried once, after infra error Harbor recipe, common settings: Attempts per task 1

Verified

  • python3 app/tests/parallel.py: 2229 tests, OK. Route suite: 547 tests, OK. The tests added with each fix fail on the code before it.
  • Stand-in provider: a 429 with Retry-After: 75 waited 75 s, then retried; a Stop during that wait or during the retry keeps the earlier try; a model the provider refuses (400) stops the run after three refusals, with no retry.
  • OpenRouter: a 6-unit run passed 6 of 6, its bill matching the relay's record; a made-up model name refused in 2 s, before any call; a capped run fetched the price list and applied the cap; with HTTPS cut off, it ran uncapped and said so.
  • Site build: every link and anchor resolves. The docs-drift check's one finding is on main too.

Risk

  • Nothing is retried by default: a rate-limited unit used to run once more (feat(standard): pace a run after a rate limit and run each rate-limited unit once more #90) and now needs --retries 1. A new Harbor recipe retries nothing (3 before); a saved recipe keeps its value.
  • A run now asks the provider for its model list before it starts, and a capped run fetches the price list first: a slow list delays the start; one that cannot be read checks nothing.
  • --repeat keeps working; recipes now write --attempts.
  • Undo: revert the squash commit.

…ith pass@k and pass^k

Trials of one task in one job are its attempts, numbered by when they
started (nullius.runs.attempts). The trial page gets a tab per attempt and
an All attempts table; the job page shows one row per task with its pass
count, the instruction score once and the mean tokens and times; the run
page and Compare show pass@k and pass^k. Compare judges a task by its pass
rate over every attempt, ranks by how far it moved (read with one pass and
one fail added, so a single flipped attempt ranks below a steady gap), and
its side-by-side page gives each run its own attempt tabs.

Attempts are read as the adjusted rate reads trials: an infra attempt is
counted apart and left out of the pass count, the means and pass@k; an
agent's timeout or crash counts as a failure. The standard's pass_hat now
calls the same estimators; what it writes is unchanged.
…nt-sorting

# Conflicts:
#	app/static/style.css
A repeat is one run of a task, counted whatever its result (Harbor's
n_attempts, the standard's --repeat). An attempt is one try within a
repeat, taken again only when the infrastructure or the agent fails, and
only its last try is kept as the repeat's trial. The pages, Compare and its
command's JSON, the field guide and the module (nullius.runs.repeats) now
say repeat where they meant one. The standard still writes pass_hat_k's
`attempts` key, the name result.json has always carried.
…tructure or the agent fails

A repeat is one run of a unit, counted whatever its result (--repeat); an
attempt is one try within a repeat. A repeat that ends in a failure a
further try may mend (Harbor's retry rule: a rate limit, a provider,
network or setup error, an agent crash, an output-length or verifier
error) runs again after the other units, one at a time and paced, until it
ends otherwise or has had --attempts tries, 2 by default: the single re-run
of a rate-limited unit, extended to every retryable failure. Never a
timeout, a refusal, a wrong model or key, a missing secret, a usage limit,
the spending cap, a stop, or a task the agent failed. Each earlier try is
kept in retried/<trial>.attempt-N; the kept trial records its attempt and
the config records attempts.

Both runners' recipes carry repeat and attempts: a Harbor recipe is saved
as version 2 (repeat is --n-attempts, attempts is --max-retries + 1) and
version 1 is read as it was saved. Where "attempt" named a repeat it now
says repeat: the builder's labels, budgets, benchmark notes, and the
driver protocol, which sends repeat to a definition declaring
limits.repeats and attempt to one still declaring limits.attempts.
@github-actions

This comment has been minimized.

…the run does

An attempt is one run of a task, counted whatever its result (Harbor's
n_attempts, the standard's --repeat); a retry is a further try of an
attempt after an infrastructure or agent failure, and only its last try is
kept as the attempt's trial. This takes back the word repeat for the first,
as Harbor names it, and the field guide's section says both.

A task's attempts are numbered by the run's own number when every trial
carries one (the standard's ..._u0042r2), so an attempt retried after the
others keeps its place; by start time otherwise, as Harbor's names carry no
number.
…the agent, with --retries beside --attempts

Merges #146 and rebuilds this change in its words. An attempt is one run
of a unit, counted whatever its result: --attempts K, the older --repeat K
still read. A retry runs an attempt again after a failure a further try
may mend, by Harbor's retry rule (a rate limit, a provider, network or
setup error, an agent crash, an output-length or verifier error; never a
timeout, a refusal, a wrong model or key, a missing secret, a usage limit,
the spending cap, a stop, or a task the agent failed): --retries N, 0 by
default, so the single re-run of a rate-limited unit now needs --retries 1.

Earlier tries are kept in retried/<trial>.try-N, the kept trial records
its retries (standard.retries) and the config records retries. The builder
says Attempts per task and Retries per attempt for both runners, and a
Harbor recipe's retries default to 0. A retried attempt says so on the run
page, the trial page, All attempts and the live board, and the progress
counts every try. The driver protocol, the catalogue and the recipe formats
keep main's words.
@github-actions

This comment has been minimized.

@aitianshi aitianshi changed the title feat(standard): add --attempts, trying a repeat again when the infrastructure or the agent fails feat(standard): retry an attempt that fails on the infrastructure or the agent, with --retries beside --attempts Nov 16, 2026
…at a further try may mend

A retried attempt's tries are tabs of their own on its trial page (Try 1,
Try 2, the last counted), and a faded row each in All attempts; the run
page and All attempts count each attempt's retries against the run's limit.

A retry follows only a failure a second try may mend: never a wrong key or
model, spent credit, a time run out, a setup the machine cannot give, or an
attempt that spent its own cap (the cap counts all of its tries together).
A rate limit's retry waits the provider's pause, or a minute. A Stop during
a retry keeps the try before; a merge keeps the shards' tries; a host run's
copy, a judgement, the progress and the live board each read the retry as
its own try. Proof runs retry a rate limit once.
…hanges a reading

Keep main's pass@k and pass^k text; add only the words for attempt and retry, which tasks the pages leave out, how a task's pass rate counts and when a Compare row disagrees.
…empts

Brings in main through #146. The conflicts are resolved as in the rebased branch the suites passed on; the resolution also cuts the retries docs and the --retries help to the AGENTS.md documentation rule.
… and say the pause in whole seconds

A rate-limited attempt stays on the run page while its retry waits, and a
Stop during the wait leaves nothing to put back; a stopped retry's empty
retried/ folder goes.
…ks and attempts apart in its last line

A retry round shares the run's concurrency (-j) instead of running one at a
time; a retry stopped under it still leaves the try before. The run's last
line says the tasks and, when each was attempted more than once, the
attempts beside them ("3 problems, 6 attempts"), never the attempts as tasks.
Each earlier try was paid for: the job's total cost adds what each
attempt's tries before its last recorded (Trial.retried_cost_usd), and the
live board's total keeps an attempt's cost when its retry starts.
A try's page found its try in the attempt's tries, then read them all again
for its tabs.
…m's checks included

The catalogue's notes, conform's flag and its pages, the import-agent skill
and the run's help name the flag --attempts; --repeat is still accepted.
…st, and fetch the price list a cap needs

A model the provider's list lacks would be refused on every call: the run is
refused before anything installs (a :suffix routes the model it follows; a
provider with no list is not checked). A capped run fetches the model price
list first; when it cannot be fetched and a model has no price, the run goes
on without the cap and says so, in its log and on its units. No test reaches
past this machine: tests/__init__.py refuses any other address.
…ck among the run's refusals

The standard's Attempts and Retries rows link the field guide and the run page instead of restating them; the retries table keeps which failures are tried again and drops how many run at once and where earlier tries are kept.
@github-actions

This comment has been minimized.

Base automatically changed from feat/compare-pass-count-sorting to main November 17, 2026 22:19
@github-actions

Copy link
Copy Markdown

🟡 PR judge · 7 of 8 yes · #154 ↔ #152

Merge risk 🔴 MAJOR the judge's reading of the diff; app/nullius/execution.py is in the path of every run
Priority 🔴 EXTREME the deepest path is app/nullius/execution.py
Question Verdict
Changes do what the issue asks · 8 of 8 criteria ✅ yes
Issue and pull request describe the same change ✅ yes
Description matches the diff, no more, no less ⚠️ partly
Diff stays in scope ✅ yes
Tests cover the change, or Verified says how ✅ yes
Docs follow the change ✅ yes
Risk named in the description ✅ yes
Before and after shown · 6 images · the diff touches what is visible ✅ yes
Acceptance criteria, 8 of 8 met
# Criterion Verdict
1 cli run --retries N (0 to 9, default 0) runs an attempt again, up to N times, only when it ended in a retryable failu… ✅ met (cut)
2 Retries run after the other units, one at a time and paced; nothing is retried once the run is stopped or halted, and t… ✅ met (cut)
3 Each earlier try is kept whole under the job's retried/, numbered, where no reader takes it for a trial; the kept tri… ✅ met (cut)
4 cli run --attempts K runs each unit K times; --repeat K still works. ✅ met (cut)
5 A standard recipe carries retries; a Harbor recipe's retries default to 0; the builder labels the settings "Attempts … ✅ met (cut)
6 A retried attempt says so on the run page, the trial page, All attempts and the live board. ✅ met (cut)
7 docs/standard.md, docs/cli/run.md, docs/builder.md and the field guide describe both settings. ✅ met (cut)
8 Tests cover the retry rule, the rounds of retries, the kept tries, a stop, the recipe keys and the pages. ✅ met (cut)
Checks the script runs itself, 4 of 4 pass
No machine attribution trailer in the commits or the description ✓
New code under app/, docs/, .github/ or .claude/ ✓
No Flask import under the backend package ✓
No key-shaped string in the diff ✓

some of the diff was cut before the judge read it

@aitianshi
aitianshi merged commit 49d32d1 into main Nov 17, 2026
2 checks passed
@aitianshi
aitianshi deleted the feat/standard-attempts branch November 17, 2026 22:20
aitianshi pushed a commit that referenced this pull request Nov 19, 2026
…viders, set its own limits and prove a merged null control (#133)

## What

A benchmark of the standard can now say more about how its units run and
end, which a computer-use benchmark like OSWorld2 (#134) needs. `gated`,
the new limits and `prove`'s new options are opt-in. The relay's rules
and the rules of how a unit ends change for every benchmark of the
standard; Risk lists them.

- **Gated tasks** (`gated: true`): a unit's failure that the relay's
records explain, a model endpoint's (the agent's, the judge's or the
simulated user's), says in its `result.json` the status and class alone,
never the provider's words, which can quote the request. The trial's
call records keep the provider's words, and an error a driver raises
itself is kept as the driver words it.
- **A benchmark option's values by name** (`names`: {value: name}): the
builder's cards and select, Review and the benchmark's page show the
name, `about` stays its tooltip, and a recipe and the CLI keep the
value. OSWorld1's agents are named as their upstream spells them.
- **An option on its own step** (`step`): `where`, where the work runs,
on the Where step, one value as fixed, and in Review's Where row;
`dataset`, what some datasets need, on the Dataset step for those
datasets. OSWorld1's VM provider is on Where.
- **Limits of a benchmark's own:**
- `limits.score_seconds`: how long `score` may take, the benchmark's
level below `--timeout score=` (#191); the Limits step shows it as the
benchmark's;
- `limits.call_seconds`: how long the relay waits for one model call,
also `relay serve --timeout`;
- **How a unit ends:**
- a driver's error that begins `EnvironmentSetupError: `,
`VerifierError: `, `CancelledError: ` or `CauseUnknownError: ` keeps
that kind;
- the agent's own last call refused as longer than the model's context
window is `ContextWindowExceededError`, the agent's;
- a last 429 is read from the agent's own calls in the order they began,
so a side model's later 200 does not cover it;
- a unit a guard's stop ended keeps the score its driver gave, its own
last call's 429 notwithstanding, unless that 429 said the credit is
spent. `score` can say the stop's reason (`verifier_result.stopped`);
- with #154's `--retries`, a unit a guard asked to stop and a full
context window are not tried again;
- for an upstream's own answerer: `OutputLengthExceededError` only when
its last reply has neither text nor a tool call. When the answerer fails
and its last reply was a Responses object answered 200 and `failed`,
that is an API error.
- **The relay:**
- a non-streamed 200 to a model call with no reply in it reaches the
caller and the record as a 502, or the 429 or 402 of an error it held
(the record keeps `answered: 200`). #124 leaves counting an error inside
a 200 to a pull request of its own; this one takes only a 200 with no
reply in it;
- a stream cut off midway, after the caller had its 200, is recorded as
a 502, and its booking against the spending cap stays whole;
- a call refused with 400, 413 or 422 records each inline image it sent
by its declared type, the type its bytes show and its size, never a
byte;
- a `stop` file beside the relay's records: the relay notes in it an
agent's call refused for the model's context window, and that note
refuses nothing. Once a benchmark's guard writes a line of its own
there, the agent's later calls are refused unsent, while side models and
the judge's and simulated user's routes go on;
- a failed Responses object with no usage records a cost of 0, and its
booking is kept.
- **A job's name:**
- a run holds its job name until its process ends, and one under a name
another run still holds is refused before it writes anything;
- a rerun into the same job moves each rerun trial's earlier records
(`.work/relay`, `result.json`, `verifier/`, `agent/`) aside to
`.work/earlier.<ns>`.
- **prove and merge:**
- `nullius standard prove ... --bench-option NAME=VALUE` sets the
benchmark's own options for the units check and the controls;
- `--job nop=MERGED` (or `oracle=`) checks a merged launch of a control
against its bound instead of running it, and every trial's own record;
- `merge` accepts shards that differ in benchmark options and keeps each
shard's in `merged_from`;
- `merge` refuses, unless `--force`, a trial whose record names another
job, benchmark, version, control or agent, and a job itself merged
despite differences. It lists the shards absent from a merge only when
the launch has at most 1,000.
- **Smaller:**
  - a trace that cannot be built leaves `agent/trace.error.txt`;
  - an Original's trajectory keeps its tool calls;
  - each call's record keeps the request's sha256;
  - `thinking_tokens` count as reasoning;
- a setup's `comparable` can list the models whose runs compare with the
paper, and the builder's message for another model says so;
- a test checks that every file a host's snapshot takes of a bundled
entry is package data.

**Read first:** `failure`, then `_hollow` and the stop in
`Relay._forward` (`app/nullius/standard/relay.py`). Next, the outcome
rules in `Run` (`app/nullius/standard/run.py`), then `names` and `step`
in `recipe.compile` and `benchOptions` in the builder, the `--job` check
in `prove.py` and the trial check in `merge.py`. Then
`test_standard_gated.py`, `test_standard_context_window.py` and
`test_standard_controls.py`. The rest is the definitions' checks, the
CLI options, the sidecar's argument and the pages.

## Touches

the standard (`app/nullius/standard/`: `relay`, `run`, `prove`,
`definitions`, `merge`, `job`, `cli`, `container`, `recipe`, `fidelity`,
`retries`) · the builder (`app/static/builder/`, `app/web/builder.py`:
the score's default on the Limits step, an option's names, and an option
on its own step) · the benchmark page (an option's names) · OSWorld1's
definition (its agents' names, its VM provider on Where) · docs and site
(`docs/standard.md`, `docs/cli/run.md`, `docs/cli/read-jobs.md`,
`docs/builder.md`, `docs/settings/model-providers.md`)

## Verified

- At `e768c632`, on main at `db6e8f96`, on Python 3.12: backend suite
(`python3 app/tests/parallel.py`), 2310 tests in 129 modules, OK; route
suite with Node on PATH, 639 tests in 53 modules, OK; the wheel's check
(`app/packaging/verify.py`), OK.
- **An option's names and its step, in a headless Chromium, light and
dark, 1400 and 390 px,** with #134 on top: OSWorld1's agents by name in
their select, OSWorld2's in their cards; the VM provider of each as a
fixed value on the Where step, shown only while a participant that takes
it is chosen; OSWorld2's Website host suffix on the Dataset step for All
tasks, not for Offline, no services; Served by empty on any choice of
model. No page error.
- **The 12 changed test modules on main's source** (`5bcfd82d`): 9 fail,
2 pass, and 1 does not end.
- `test_standard_context_window`: 19 of 20; `test_standard_controls`: 17
of 43; `test_standard_relay`: 8 of 58; `test_standard_gated`: 4 of 6.
- `test_standard_run`: 2 of 50; `test_standard_trace`: 2 of 20;
`test_standard_parts`, `test_standard_served_by` and
`test_standard_units`: 1 each.
- `test_standard_shard` does not end within 4 minutes: one of its merges
names 2^64 shards, and main's `merge` lists every absent one.
  - `test_standard_versions` and `test_dependency_surface` pass on both.
- **Covered in #134, not here:** that `limits.call_seconds` reaches the
relay and the sidecar's `--timeout`. OSWorld2 is the first entry to set
it (`test_osworld2_muse.py`).
- **#124's own tests.** Three were changed for two deliberate
differences: a 200 with no reply is a 502, and a failed Responses object
is read as failed. The rest of `test_standard_served_by` passes as main
has it; `write_shard` in `test_standard_shard` takes #137's `reward`
beside this branch's `bench_options`.
- `PYTHONPATH=app python3 -m nullius standard check`: ok. `python3
app/tests/catalogue_docs.py --write`, then the check: up to date.
`python3 app/tests/field_contract.py`: ok. `app/site/readme_media.py
--check`: current. The docs check reports the same two findings in
`AGENTS.md` as on main.
- Not run: a benchmark already on main, live, on this branch; the
container backend with `--timeout`; the docker tier; the site build.

## Risk

- **For every benchmark of the standard:**
- a non-streamed 200 with no reply is a 502, and a stream cut off midway
is recorded as one;
- the agent's own last call refused for its context window is the
agent's error (`ContextWindowExceededError`), where it was an API error
outside its rate;
- an agent's last 429 followed by a side model's 200 makes the unit
`rate_limited`, where its score stood;
- an upstream answerer whose last reply was a `failed` Responses object
is an API error, where it was the agent's;
- a driver's error of a kind it names: a setup error or a stop is left
out of the agent's rate, and a verifier error or one of no known cause
counts as unknown. A wrong kind would move a unit between them;
- `prove` runs no control when the units check failed, and fails a
control job with a unit that has neither a score nor an error;
- a run under a job name another run still holds is refused, and a rerun
into an existing job moves the rerun trials' records aside;
- `merge` refuses a trial whose record names another job unless
`--force`.
- **Records.** Stored records gain fields and keep the old ones.
- These commits sit on main at `db6e8f96`, with #124, #150, #191, #154,
#174 and the Recipes page (#205). #124's `--served-by` code is main's as
merged; #191's time limits come first, so `--timeout score=` is used
over `limits.score_seconds`, and a unit's cap on model calls
(`UnitCallCapError`) is read before this PR's outcome rules. #154's
`--retries` replaces the one more run of a rate-limited unit, so the
earlier `limits.run_again: false` that switched it off is gone; main
counts a provider's 402 as a usage limit, and the tests follow. The
first and fourth commits gave a definition pins of its serving
providers, defaults of Served by after review; the last commit removes
them, as the reviewer asked, since no benchmark declares one: the diff
over main has none, and Served by is the user's alone, as in #124.
- The fifth commit shortens this PR's docs and messages. No logic
changes.
- **Names and Where.** An option's names change only what is shown; a
recipe saved before keeps working. An option of where the work runs
leaves the Agent step for Where, and one some datasets need is asked on
Dataset: OSWorld1's VM provider is now on Where.
- Undone by reverting the commits.

---------

Co-authored-by: Oleg <247799477+sirotaoleg@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Standard: no --retries for an attempt that fails on the infrastructure or the agent, and --repeat is not named --attempts

1 participant