Skip to content

fix(standard): time the agent's setup apart from its run - #174

Merged
aitianshi merged 8 commits into
mainfrom
fix/standard-agent-setup-phase
Nov 17, 2026
Merged

aitianshi merged 8 commits into
mainfrom
fix/standard-agent-setup-phase

Conversation

@aitianshi

@aitianshi aitianshi commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #180

What

A standard trial of an agent driven over ACP had no agent setup: the agent's process start, initialize and session/new (which also starts the benchmark's MCP servers) ran inside the window written as agent_execution. The trial page read "Setup —" and its Agent time carried the start.

The run now notes when the agent's session opens and writes Harbor's agent_setup, from the end of the environment's setup to the open session. agent_execution, and the window of the trial's trace, start there.

The second commit corrects what an independent review of the first found:

  • A start that ends with no session (the agent never answered, exited, or the run was stopped) was still written as the agent's run. It is its setup, with no agent_execution, as Harbor writes a failed agent setup.
  • A capability got a setup of a millisecond: the time to spawn a process that answers nothing until its first request. A capability, a command agent, an Original setup and a control have no setup phase.
  • A task's internet door opened inside the agent's setup and closed inside its run. The agent's phases begin once it is open and end before it closes.
  • The live feed still sent one duration, timed from before the setup, and no agent_setup phase, so the live board disagreed with the trial page. It emits the phase, starts the agent's limit when the session opens, and takes every duration from the trial's result.
  • The trace-window test could not fail. It pins the window now, and the run's side has tests of its own.

The last two commits take main (ten commits: a unit's VM, forced time limits, attempts and retries among them):

  • The unit loop reads: the unit's VM, the task's door, then the agent's setup and its run.
  • main renamed the turn's limit to the agent's whole work on a unit (agent_limit) and starts its clock once the session is open. The board's agent_run phase is emitted there with that same limit, a run's --timeout agent included; the setup keeps ACP's two start limits. The line this branch had added still called the old name and merged without a conflict: every ACP unit would have ended in an error.
  • The trial page's Setup hover is left to feat(traces): rework the trial page around what a trial recorded #173, which merges first and words it for a standard run.

Installing the agent stays a stage of the run, not of a trial.

Touches

  • the standard: standard/run.py (the unit loop, drive_agent), standard/job.py (write_trial, feed_durations), standard/acp.py (its two start limits, named)
  • docs: field-guide.md (the phase rows, shorter)
  • tests: test_standard_trace.py, test_standard_run.py

Before and after

The same recipe (agent-task-v1, the reference agent, a stand-in endpoint answering in 2 s) run on #173, which merges first, and on #173 with this branch: the trial page this lands on. Before, the agent's start sits in Agent; after, it is Setup.

Before After
Before: no Setup phase After: Setup 0.06s apart from Agent

Verified

  • On the merge with main (49d32d1): backend suite 2236 OK, route suite 547 OK. With feat(traces): rework the trial page around what a trial recorded #173 merged first: 2236 and 562 OK.
  • Seven of the new or extended tests fail on the first commit and pass on the second: the four windows on shared stamps with the feed told the same seconds; a start that never answers; a capability; a conversation the upstream starts again; the task's door (a stand-in door that takes 0.4 s to open and to close); a failed start and a trial with no clock in write_trial. The trace's window is pinned as well: it starts at the open session, and a model call made during the setup widens it and is never clamped.
  • After the merge, a new test forces --timeout agent and reads the feed: the setup carries ACP's 300 s, the run the forced limit. With the old limit's name left in, the setup test ends in an error.
  • A real local run on the merged code (nullius standard run agent-task-v1 reference --limit 1 --unsandboxed --timeout agent=45, stand-in endpoint): result.json holds environment_setup 0.046 s, agent_setup 0.054 s, agent_execution 2.056 s, verifier 0.001 s, back to back; events.jsonl holds the phases env_setup (600 s), agent_setup (300 s), agent_run (45 s), verify (600 s) and a trial.end with the same four durations; the trace's session is 2.056 s.
  • Not verified: the docker tier and the Boat backend were not run, so a real container's internet door and a unit's VM were not timed; the door's order is held by the test with the stand-in door, and the VM wraps the same block.

Risk

  • Agent time on standard runs written before and after this differs by the agent's start; a comparison across that line reads slightly different. Stored results are not rewritten, and the field guide says so.
  • On a task with an internet door, the environment's time grows by the door opening and the agent's run shrinks by its opening and closing.
  • A failed start has no agent_execution. Readers already meet a trial without one (a unit that fails in its environment's setup).
  • The live board shows "agent setup" while the agent starts, with a limit of 300 s (ACP's two start requests), then the agent's limit from its open session.
  • Merges after feat(traces): rework the trial page around what a trial recorded #173. Merged alone, the Setup hover on a standard trial would still read "installing the agent" until feat(traces): rework the trial page around what a trial recorded #173 lands.
  • Undone by a revert.

A unit's agent started and opened its session inside agent_execution, so
the trial page had no Setup phase and Agent time carried the handshake
and the benchmark's MCP servers starting. The run now notes when the
session opened (an ACP agent) or the process started (a capability) and
writes it as Harbor's agent_setup; agent_execution and the trace's window
start there. A command agent and an upstream setup start and run in one
call, so they have none.
@github-actions

This comment has been minimized.

… and tell the board the same

Review of the first commit found the trial's result and the live board
disagreeing, and three windows that were not what they said:

- a start that ended with no session (never answered, exited, stopped)
  was written as the agent's run; it is now its setup, with no run, as
  Harbor writes a failed agent setup;
- a capability got a setup of a millisecond, the time to spawn its
  process, which answers nothing until its first request; it has none now;
- the task's internet door opened inside the agent's setup and closed
  inside its run; the agent now begins once it is open and ends before it
  closes;
- the live feed still sent one duration, timed from before the setup, and
  no agent_setup phase; it now sends the phase, starts the turn's limit
  when the session opens, and takes every duration from the result.

The trace-window test could not fail (its events began before the setup);
it now pins the window, and the run's side has tests of its own: the four
windows on shared stamps, a failed start, a capability, a conversation
started again, the door. The field guide says which runs have a setup and
what Other is on a standard run, and the trial page's hover no longer
calls a standard agent's setup an installation.
@github-actions

This comment has been minimized.

aitianshi added a commit that referenced this pull request Nov 17, 2026
… work on a unit and cap its model calls (#191)

Closes #190

## What

A standard run can now force its limits, and the agent's time limit
covers its whole work on a unit.

- **Time limits by phase.** `run --timeout PHASE=SECONDS` forces the
limit of a phase: `setup`, `image`, `agent`, `verify` or `score`. Empty
keeps the default, which comes from the task, then the benchmark, then
Nullius. `config.json` records what was forced under
`standard.timeouts`.
- **One agent limit per unit.** The agent's limit is one deadline for
its whole work on a unit: every turn of a conversation, and the
simulated user's time between them. Before, it applied to each reply. A
capability keeps one limit per add and per recall, because a memory adds
a unit's sessions one by one. τ³-bench's `limits.turn_seconds` goes from
600 s a reply to 7200 s a conversation.
- **Turns.** `run --max-turns N` sets the turns of a conversation. The
builder offers it only on a benchmark that holds conversations.
- **A cap on model calls.** `run --max-calls N` caps the model calls an
agent makes on a unit. The relay counts each answered call on the
agent's route; rate-limited and failed calls are not counted, and
neither are the judge's or the simulator's. Past the cap the relay
refuses the call with a 402, and the unit ends as `UnitCallCapError`, an
agent error that is not scored. It is for agents only: a capability's
route also carries the benchmark's reader, so a capability run refuses
it.
- **A benchmark's own bounds.** They are now options marked `limit`.
τ³-bench gets Steps per conversation (`--max-steps`, from 1 up to its
default of 200) and Tool errors per conversation (`--max-errors`).
OSWorld gets Steps per task (each agent script's `--max_steps`); left
empty it uses the chosen agent's own value, declared with the new `by`
key.
- **Builder, Limits step.** It shows Scope, then Time limits in minutes,
then a section named after the benchmark when it has bounds of its own.
An empty field shows the value the run uses: in the text colour when the
benchmark or the agent declares it, grey when it is Nullius's default
(priority: Nullius < agent < benchmark). Nothing is typed into a field
for the user.

## Touches

- `app/nullius/standard/`: `run.py`, `relay.py`, `container.py`,
`job.py`, `cli.py`, `recipe.py`, `definitions.py`
- `app/nullius/runs/classify.py`: `UnitCallCapError` is an agent error
- Catalogue: τ³-bench and OSWorld's `standard.json`, their drivers and
`upstream-settings.json`
- `app/static/builder/`: the Limits step
- Docs: `docs/cli/run.md`, `docs/standard.md`, `docs/builder.md`, the
field guide and the generated supported-benchmark pages

## Before and after

The Limits step for τ³-bench, before and after:

| Before | After |
| --- | --- |
| <img width="440" alt="Limits step on main: Scope only, with tasks,
concurrency, attempts and two spending caps"
src="https://github.com/user-attachments/assets/80b468a4-326f-4057-a4b2-5551ea93e642"
/> | <img width="440" alt="Limits step on this branch: Scope with turns
and model calls, Time limits in minutes, then a tau3-bench section with
steps and tool errors"
src="https://github.com/user-attachments/assets/6278494c-6c9f-456f-b739-9578494315ee"
/> |

## Verified

- Backend suite: 2175 tests OK. Route suite: 531 tests OK (after merging
`main`, with #124's serving providers and #150's Boat VMs).
- New tests:
- A whole conversation that ends past the agent limit, though no single
turn does.
  - Four one-call turns cut on the fourth call by `max_calls=3`.
  - The relay counts only answered calls on the agent's route.
- A slow capability whose three adds each fit a 1 s limit while their
total does not.
  - A capability run refuses `--max-calls`.
- Recipe validation of the limits, including a value that rounds to 0 s.
  - τ³-bench steps above 200 are refused.
- OSWorld's `--max_steps` is replaced, or added when the agent's command
doesn't set it.
- Against the installed τ³-bench upstream: the driver uses 200 steps and
10 errors by default, and 7 and 3 when the run sets them.
- Checked in the browser, light and dark, for τ³-bench and OSWorld.

## Risk

- **Agent timeouts on conversation benchmarks.** A conversation that
used to fit 600 s per reply now has a single limit for the whole
conversation, 7200 s on τ³-bench. A very long conversation can now time
out where it didn't before. Undo by forcing a larger `--timeout
agent=SECONDS`, or by reverting `limits.turn_seconds`.
- **Comparability.** A run whose limits differ from the defaults is not
yet marked as not comparable with the leaderboard.
- **Overlap.** #174 (the agent's setup timed apart from its run) and
#133 (a benchmark's own limits) change the same parts of `run.py` and
the catalogue, so expect merge conflicts.
Ten commits landed on main since this branch left it; the unit loop and the
trial writer conflicted.

- standard/run.py, the unit loop: main runs the agent inside the unit's VM
  (unit_vm) and names the limits anew (agent_limit, phase_limit); this branch
  opens the task's door before the agent's phases and closes it after them.
  The VM, then the door, then the agent's setup and run, on main's limits.
- standard/run.py, drive_agent: main dropped turn_limit for agent_limit, and
  starts the agent's clock once its session is open. The board's agent_run
  phase is emitted there with that same limit; the line this branch added
  still called turn_limit, which git merged without a conflict.
- standard/run.py, unit(): over_acp() stays, with main's retry number.
- standard/job.py, write_trial: both sides added parameters; all stay.
…ere it starts

main lets a run force the agent's limit (--timeout agent). The board is told that limit for the agent's run, from its open session, and ACP's two start limits for its setup; the run command's page says an ACP agent starts before its limit does.
@github-actions

This comment has been minimized.

#173 merges first and words each phase's hover by the kind of run, a standard run's setup included; this branch's own wording of the same line would only conflict with it.
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

aitianshi added a commit that referenced this pull request Nov 17, 2026
Closes #179

## What

A trial's page spent its first screen on parts that said nothing or said
it twice, and hid some of what the trial recorded. This reworks it into
a summary of the trial.

- **Head.** The name is smaller. One row under it says what ran (the
benchmark and its runner, or the importer of an imported trial; the
agent with its mark, name, version and model) and how it went (the
dataset's checks, the reward or the score, the agent's time, the steps).
The Tool calls card and the evidence strip (Trajectory, Trace, Reward,
Rate limits) are gone. A value the trial has not has no cell and no
dash.
- **Overview.** "Shape of the run" is gone. "Where the time went" is a
compact tile in a grid that other tiles can join: its total in the
heading, no paragraph, a chip for every phase the trial recorded however
short, each with a hover saying what the phase holds on that kind of
run. A panel with nothing to show is not drawn.
- **Durations.** Two decimals wherever one is drawn: the `secs` filter
(every page that uses it), the run page's script, a span's minutes
(`nsdur`), `nullius runs show`. One decimal turned 0.05 s into "0.1s".
- **Artifacts.** The tab lists only the records the trial wrote, each
with Read and Download. A record's page has Copy and Download of the
whole record, redacted as the reader shows it
(`/trial/…/artifact/<key>/download`), and a back button to the Artifacts
tab. A record past the reader's 16 MiB is saved as
`<name>.partial.<ext>`.
- **Tabs.** Artifacts, Conversation and the judge tabs no longer repeat
their tab's name as a heading. The judge tabs keep the experimental
badge (who answered, the question set, the date and the tokens in its
hover) and say a refusal, an earlier vocabulary or a truncated state
only when there is one.
- **Removed.** The "benchmark's own judge" card, with the code that read
`verifier/judge.json`: nothing in the application ever wrote that file.
Its two hand-made fixture copies are deleted with it, which is why the
judge's script floors this at severe.

Commits nine and ten correct what an independent review of the first
eight found: cells colliding in the head at mid widths, sideways scroll
at phone width, an imported trial said to be Harbor's, dashes where the
page claimed none, three places still at one decimal, a silent cut of
large downloads, Copy saying copied when a browser refused the write,
and tests whose names no longer matched what they asserted.

The last three take `main` (ten commits, among them a task's attempts
and an attempt's tries, each with tabs on this page):

- The head reads: the name, what ran, then the attempt and try tabs,
then that attempt's numbers. What is above the tabs is every attempt's,
so a tab changes what is under it and the tabs stay put. A trial with no
tab keeps its one row.
- All attempts has the same head (one macro, `traces/_head.html`) in the
place of its large name and subtitle.
- Download now reads an earlier try's records, as the record page does;
it answered 404 there.

## Touches

- templates and static: `traces/trial.html`, `traces/_head.html` (new),
`traces/attempts.html`, `_judgement.html`, `_conversation.html`,
`artifact.html`, `runs/_execution.html` (its script's `secs`),
`base.html` (the copy handler), `style.css`, `traces/conversation.css`
- routes: `web/traces.py` (the download route, the importer's name),
`web/filters.py` (`secs`, `nsdur`), `web/judge.py`
- backend: `runs/artifacts.py` (`whole`), `runs/cli.py` (two decimals)
- docs: `field-guide.md` (`n_rate_limited_retries`), `cli/pages.md` (the
download route), the agent marks README
- tests: route tests (`test_attempt_routes.py` among them), and a new
`ui_tests/test_trial_head.py`

## Before and after

`main` and this branch serving the same job trees, dark, 1280 px wide,
rail collapsed.

**Overview, a SWE-bench trial**

| Before | After |
| --- | --- |
| <img width="420" alt="Before: cards, evidence strip, title, bar and
paragraph"
src="https://github.com/user-attachments/assets/1f59e693-1bb7-47a8-8407-3202eb96527a"
/> | <img width="420" alt="After: one row of blocks and a time tile"
src="https://github.com/user-attachments/assets/b9f34372-0954-4648-a885-84b647b9eb0d"
/> |

**Artifacts tab**

| Before | After |
| --- | --- |
| <img width="420" alt="Before: nine rows, six of them not recorded"
src="https://github.com/user-attachments/assets/f4b3f4fe-fe79-4248-90ec-c5a25b7bd534"
/> | <img width="420" alt="After: the three records written, each with
Read and Download"
src="https://github.com/user-attachments/assets/7b1ffcab-ab3d-48b4-938f-cd1e2351f475"
/> |

**A record**

| Before | After |
| --- | --- |
| <img width="420" alt="Before: a back link and the page"
src="https://github.com/user-attachments/assets/c7207091-803e-4a45-9558-16d6242b30de"
/> | <img width="420" alt="After: a back button, Copy and Download"
src="https://github.com/user-attachments/assets/689dc2bd-d7cf-4f92-902f-c87b05945080"
/> |

**Judge tab**

| Before | After |
| --- | --- |
| <img width="420" alt="Before: the benchmark judge card, a heading and
a description"
src="https://github.com/user-attachments/assets/ee428219-8582-4699-9847-d3f2cf11383a"
/> | <img width="420" alt="After: the badge and the cards"
src="https://github.com/user-attachments/assets/20365bea-f70d-4349-8552-af91fc62becf"
/> |

## Verified

- On the merge with `main` (`49d32d1`): backend suite 2229 OK, route
suite 562 OK. With #174 merged on top as well: 2236 and 562 OK.
- New tests: the head for a Harbor trial, a trial with nothing to show,
a SWE-bench trial with and without its report, an imported trial, a
scored trial; the cut download; the judge tab's notes; a span's minutes
against `secs`; the run page's script against the template (run in
node); a bar shorter than a second; the shell's phases against the
page's.
- Attempts: a retried attempt's earlier try downloads its own record
(the test fails without the fix); the attempt and try tabs sit at the
same height on every attempt of a task and on All attempts, at 820, 1024
and 1280 px, with no sideways scroll or clipped text at 390.
- Browser, measured in the page: a six-cell SWE-bench trial at 390, 820,
1024 and 1280 px has no cell wider than its box and no sideways scroll;
the Artifacts tab and a three-page record at 390 px fit; dark and light
at 1280 px.
- Copy: the whole record reaches the clipboard (22,990 characters of a
record shown 8,000 a page) through each of its three paths, checked with
the clipboard stubbed, and a real click reads "copied".
- An independent review of the first eight commits: twelve findings,
each fixed or answered in the last two commits.
- Not verified: Safari (the Copy path written for it could not be run
here), the judge tabs with a real judgement (a seeded record in a test),
the nightly UI sweep.

## Risk

- Every duration drawn through `secs` gains a second decimal, on every
page that uses it. A stage time recorded to one decimal reads "41.20s".
- The trial page no longer shows the count of retried rate limits; the
job table's `429 ×N` badge keeps it.
- Deleting the two `judge.json` fixtures removes the only trace of a
card nothing could fill; the judge's script floors a deleted test file
at severe.
- Merges before #174, which no longer touches `traces/trial.html`: the
two merge cleanly in that order.
- Undone by a revert: no stored data changes shape.
@github-actions

Copy link
Copy Markdown

🟡 PR judge · 8 of 8 yes · #174 ↔ #180

Merge risk 🔴 MAJOR the judge's reading of the diff; app/nullius/standard/acp.py is in the path of every run
Priority 🔴 EXTREME the deepest path is app/nullius/standard/acp.py
Question Verdict
Changes do what the issue asks · 6 of 6 criteria ✅ yes
Issue and pull request describe the same change ✅ yes
Description matches the diff, no more, no less ✅ yes
Diff stays in scope ✅ yes
Tests cover the change, or Verified says how ✅ yes
Docs follow the change ✅ yes
Risk named in the description ✅ yes
Before and after shown · 2 images · the diff touches what is visible ✅ yes
Acceptance criteria, 6 of 6 met
# Criterion Verdict
1 A standard trial of an agent driven over ACP records agent_setup, from the end of its environment's setup to its sess… ✅ met
2 A start that ends with no session (the agent never answered, exited, or the run was stopped) is recorded as setup, with… ✅ met
3 A command agent, a capability, an Original setup and a control record no setup. ✅ met
4 A task's internet door opens before the agent's phases and closes after them: neither is in agent_setup or `agent_exe… ✅ met
5 The live feed emits an agent_setup phase for such an agent, and its trial.end durations are the trial result's wind… ✅ met
6 The field guide's phase rows hold for every runner, not Harbor alone. ✅ met
Checks the script runs itself, 4 of 4 pass
No machine attribution trailer in the commits or the description ✓
New code under app/, docs/, .github/ or .claude/ ✓
No Flask import under the backend package ✓
No key-shaped string in the diff ✓

@aitianshi
aitianshi merged commit 9bc4b12 into main Nov 17, 2026
2 checks passed
@aitianshi
aitianshi deleted the fix/standard-agent-setup-phase branch November 17, 2026 23:48
aitianshi pushed a commit that referenced this pull request Nov 19, 2026
…viders, set its own limits and prove a merged null control (#133)

## What

A benchmark of the standard can now say more about how its units run and
end, which a computer-use benchmark like OSWorld2 (#134) needs. `gated`,
the new limits and `prove`'s new options are opt-in. The relay's rules
and the rules of how a unit ends change for every benchmark of the
standard; Risk lists them.

- **Gated tasks** (`gated: true`): a unit's failure that the relay's
records explain, a model endpoint's (the agent's, the judge's or the
simulated user's), says in its `result.json` the status and class alone,
never the provider's words, which can quote the request. The trial's
call records keep the provider's words, and an error a driver raises
itself is kept as the driver words it.
- **A benchmark option's values by name** (`names`: {value: name}): the
builder's cards and select, Review and the benchmark's page show the
name, `about` stays its tooltip, and a recipe and the CLI keep the
value. OSWorld1's agents are named as their upstream spells them.
- **An option on its own step** (`step`): `where`, where the work runs,
on the Where step, one value as fixed, and in Review's Where row;
`dataset`, what some datasets need, on the Dataset step for those
datasets. OSWorld1's VM provider is on Where.
- **Limits of a benchmark's own:**
- `limits.score_seconds`: how long `score` may take, the benchmark's
level below `--timeout score=` (#191); the Limits step shows it as the
benchmark's;
- `limits.call_seconds`: how long the relay waits for one model call,
also `relay serve --timeout`;
- **How a unit ends:**
- a driver's error that begins `EnvironmentSetupError: `,
`VerifierError: `, `CancelledError: ` or `CauseUnknownError: ` keeps
that kind;
- the agent's own last call refused as longer than the model's context
window is `ContextWindowExceededError`, the agent's;
- a last 429 is read from the agent's own calls in the order they began,
so a side model's later 200 does not cover it;
- a unit a guard's stop ended keeps the score its driver gave, its own
last call's 429 notwithstanding, unless that 429 said the credit is
spent. `score` can say the stop's reason (`verifier_result.stopped`);
- with #154's `--retries`, a unit a guard asked to stop and a full
context window are not tried again;
- for an upstream's own answerer: `OutputLengthExceededError` only when
its last reply has neither text nor a tool call. When the answerer fails
and its last reply was a Responses object answered 200 and `failed`,
that is an API error.
- **The relay:**
- a non-streamed 200 to a model call with no reply in it reaches the
caller and the record as a 502, or the 429 or 402 of an error it held
(the record keeps `answered: 200`). #124 leaves counting an error inside
a 200 to a pull request of its own; this one takes only a 200 with no
reply in it;
- a stream cut off midway, after the caller had its 200, is recorded as
a 502, and its booking against the spending cap stays whole;
- a call refused with 400, 413 or 422 records each inline image it sent
by its declared type, the type its bytes show and its size, never a
byte;
- a `stop` file beside the relay's records: the relay notes in it an
agent's call refused for the model's context window, and that note
refuses nothing. Once a benchmark's guard writes a line of its own
there, the agent's later calls are refused unsent, while side models and
the judge's and simulated user's routes go on;
- a failed Responses object with no usage records a cost of 0, and its
booking is kept.
- **A job's name:**
- a run holds its job name until its process ends, and one under a name
another run still holds is refused before it writes anything;
- a rerun into the same job moves each rerun trial's earlier records
(`.work/relay`, `result.json`, `verifier/`, `agent/`) aside to
`.work/earlier.<ns>`.
- **prove and merge:**
- `nullius standard prove ... --bench-option NAME=VALUE` sets the
benchmark's own options for the units check and the controls;
- `--job nop=MERGED` (or `oracle=`) checks a merged launch of a control
against its bound instead of running it, and every trial's own record;
- `merge` accepts shards that differ in benchmark options and keeps each
shard's in `merged_from`;
- `merge` refuses, unless `--force`, a trial whose record names another
job, benchmark, version, control or agent, and a job itself merged
despite differences. It lists the shards absent from a merge only when
the launch has at most 1,000.
- **Smaller:**
  - a trace that cannot be built leaves `agent/trace.error.txt`;
  - an Original's trajectory keeps its tool calls;
  - each call's record keeps the request's sha256;
  - `thinking_tokens` count as reasoning;
- a setup's `comparable` can list the models whose runs compare with the
paper, and the builder's message for another model says so;
- a test checks that every file a host's snapshot takes of a bundled
entry is package data.

**Read first:** `failure`, then `_hollow` and the stop in
`Relay._forward` (`app/nullius/standard/relay.py`). Next, the outcome
rules in `Run` (`app/nullius/standard/run.py`), then `names` and `step`
in `recipe.compile` and `benchOptions` in the builder, the `--job` check
in `prove.py` and the trial check in `merge.py`. Then
`test_standard_gated.py`, `test_standard_context_window.py` and
`test_standard_controls.py`. The rest is the definitions' checks, the
CLI options, the sidecar's argument and the pages.

## Touches

the standard (`app/nullius/standard/`: `relay`, `run`, `prove`,
`definitions`, `merge`, `job`, `cli`, `container`, `recipe`, `fidelity`,
`retries`) · the builder (`app/static/builder/`, `app/web/builder.py`:
the score's default on the Limits step, an option's names, and an option
on its own step) · the benchmark page (an option's names) · OSWorld1's
definition (its agents' names, its VM provider on Where) · docs and site
(`docs/standard.md`, `docs/cli/run.md`, `docs/cli/read-jobs.md`,
`docs/builder.md`, `docs/settings/model-providers.md`)

## Verified

- At `e768c632`, on main at `db6e8f96`, on Python 3.12: backend suite
(`python3 app/tests/parallel.py`), 2310 tests in 129 modules, OK; route
suite with Node on PATH, 639 tests in 53 modules, OK; the wheel's check
(`app/packaging/verify.py`), OK.
- **An option's names and its step, in a headless Chromium, light and
dark, 1400 and 390 px,** with #134 on top: OSWorld1's agents by name in
their select, OSWorld2's in their cards; the VM provider of each as a
fixed value on the Where step, shown only while a participant that takes
it is chosen; OSWorld2's Website host suffix on the Dataset step for All
tasks, not for Offline, no services; Served by empty on any choice of
model. No page error.
- **The 12 changed test modules on main's source** (`5bcfd82d`): 9 fail,
2 pass, and 1 does not end.
- `test_standard_context_window`: 19 of 20; `test_standard_controls`: 17
of 43; `test_standard_relay`: 8 of 58; `test_standard_gated`: 4 of 6.
- `test_standard_run`: 2 of 50; `test_standard_trace`: 2 of 20;
`test_standard_parts`, `test_standard_served_by` and
`test_standard_units`: 1 each.
- `test_standard_shard` does not end within 4 minutes: one of its merges
names 2^64 shards, and main's `merge` lists every absent one.
  - `test_standard_versions` and `test_dependency_surface` pass on both.
- **Covered in #134, not here:** that `limits.call_seconds` reaches the
relay and the sidecar's `--timeout`. OSWorld2 is the first entry to set
it (`test_osworld2_muse.py`).
- **#124's own tests.** Three were changed for two deliberate
differences: a 200 with no reply is a 502, and a failed Responses object
is read as failed. The rest of `test_standard_served_by` passes as main
has it; `write_shard` in `test_standard_shard` takes #137's `reward`
beside this branch's `bench_options`.
- `PYTHONPATH=app python3 -m nullius standard check`: ok. `python3
app/tests/catalogue_docs.py --write`, then the check: up to date.
`python3 app/tests/field_contract.py`: ok. `app/site/readme_media.py
--check`: current. The docs check reports the same two findings in
`AGENTS.md` as on main.
- Not run: a benchmark already on main, live, on this branch; the
container backend with `--timeout`; the docker tier; the site build.

## Risk

- **For every benchmark of the standard:**
- a non-streamed 200 with no reply is a 502, and a stream cut off midway
is recorded as one;
- the agent's own last call refused for its context window is the
agent's error (`ContextWindowExceededError`), where it was an API error
outside its rate;
- an agent's last 429 followed by a side model's 200 makes the unit
`rate_limited`, where its score stood;
- an upstream answerer whose last reply was a `failed` Responses object
is an API error, where it was the agent's;
- a driver's error of a kind it names: a setup error or a stop is left
out of the agent's rate, and a verifier error or one of no known cause
counts as unknown. A wrong kind would move a unit between them;
- `prove` runs no control when the units check failed, and fails a
control job with a unit that has neither a score nor an error;
- a run under a job name another run still holds is refused, and a rerun
into an existing job moves the rerun trials' records aside;
- `merge` refuses a trial whose record names another job unless
`--force`.
- **Records.** Stored records gain fields and keep the old ones.
- These commits sit on main at `db6e8f96`, with #124, #150, #191, #154,
#174 and the Recipes page (#205). #124's `--served-by` code is main's as
merged; #191's time limits come first, so `--timeout score=` is used
over `limits.score_seconds`, and a unit's cap on model calls
(`UnitCallCapError`) is read before this PR's outcome rules. #154's
`--retries` replaces the one more run of a rate-limited unit, so the
earlier `limits.run_again: false` that switched it off is gone; main
counts a provider's 402 as a usage limit, and the tests follow. The
first and fourth commits gave a definition pins of its serving
providers, defaults of Served by after review; the last commit removes
them, as the reviewer asked, since no benchmark declares one: the diff
over main has none, and Served by is the user's alone, as in #124.
- The fifth commit shortens this PR's docs and messages. No logic
changes.
- **Names and Where.** An option's names change only what is shown; a
recipe saved before keeps working. An option of where the work runs
leaves the Agent step for Where, and one some datasets need is asked on
Dataset: OSWorld1's VM provider is now on Where.
- Undone by reverting the commits.

---------

Co-authored-by: Oleg <247799477+sirotaoleg@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Standard: a trial records no agent setup, and its agent time carries the agent's start

1 participant