Repository navigation
fix(standard): time the agent's setup apart from its run - #174
Merged
Merged
Conversation
A unit's agent started and opened its session inside agent_execution, so the trial page had no Setup phase and Agent time carried the handshake and the benchmark's MCP servers starting. The run now notes when the session opened (an ACP agent) or the process started (a capability) and writes it as Harbor's agent_setup; agent_execution and the trace's window start there. A command agent and an upstream setup start and run in one call, so they have none.
This comment has been minimized.
This comment has been minimized.
… and tell the board the same Review of the first commit found the trial's result and the live board disagreeing, and three windows that were not what they said: - a start that ended with no session (never answered, exited, stopped) was written as the agent's run; it is now its setup, with no run, as Harbor writes a failed agent setup; - a capability got a setup of a millisecond, the time to spawn its process, which answers nothing until its first request; it has none now; - the task's internet door opened inside the agent's setup and closed inside its run; the agent now begins once it is open and ends before it closes; - the live feed still sent one duration, timed from before the setup, and no agent_setup phase; it now sends the phase, starts the turn's limit when the session opens, and takes every duration from the result. The trace-window test could not fail (its events began before the setup); it now pins the window, and the run's side has tests of its own: the four windows on shared stamps, a failed start, a capability, a conversation started again, the door. The field guide says which runs have a setup and what Other is on a standard run, and the trial page's hover no longer calls a standard agent's setup an installation.
This was referenced Nov 17, 2026
This comment has been minimized.
This comment has been minimized.
aitianshi
added a commit
that referenced
this pull request
Nov 17, 2026
… work on a unit and cap its model calls (#191) Closes #190 ## What A standard run can now force its limits, and the agent's time limit covers its whole work on a unit. - **Time limits by phase.** `run --timeout PHASE=SECONDS` forces the limit of a phase: `setup`, `image`, `agent`, `verify` or `score`. Empty keeps the default, which comes from the task, then the benchmark, then Nullius. `config.json` records what was forced under `standard.timeouts`. - **One agent limit per unit.** The agent's limit is one deadline for its whole work on a unit: every turn of a conversation, and the simulated user's time between them. Before, it applied to each reply. A capability keeps one limit per add and per recall, because a memory adds a unit's sessions one by one. τ³-bench's `limits.turn_seconds` goes from 600 s a reply to 7200 s a conversation. - **Turns.** `run --max-turns N` sets the turns of a conversation. The builder offers it only on a benchmark that holds conversations. - **A cap on model calls.** `run --max-calls N` caps the model calls an agent makes on a unit. The relay counts each answered call on the agent's route; rate-limited and failed calls are not counted, and neither are the judge's or the simulator's. Past the cap the relay refuses the call with a 402, and the unit ends as `UnitCallCapError`, an agent error that is not scored. It is for agents only: a capability's route also carries the benchmark's reader, so a capability run refuses it. - **A benchmark's own bounds.** They are now options marked `limit`. τ³-bench gets Steps per conversation (`--max-steps`, from 1 up to its default of 200) and Tool errors per conversation (`--max-errors`). OSWorld gets Steps per task (each agent script's `--max_steps`); left empty it uses the chosen agent's own value, declared with the new `by` key. - **Builder, Limits step.** It shows Scope, then Time limits in minutes, then a section named after the benchmark when it has bounds of its own. An empty field shows the value the run uses: in the text colour when the benchmark or the agent declares it, grey when it is Nullius's default (priority: Nullius < agent < benchmark). Nothing is typed into a field for the user. ## Touches - `app/nullius/standard/`: `run.py`, `relay.py`, `container.py`, `job.py`, `cli.py`, `recipe.py`, `definitions.py` - `app/nullius/runs/classify.py`: `UnitCallCapError` is an agent error - Catalogue: τ³-bench and OSWorld's `standard.json`, their drivers and `upstream-settings.json` - `app/static/builder/`: the Limits step - Docs: `docs/cli/run.md`, `docs/standard.md`, `docs/builder.md`, the field guide and the generated supported-benchmark pages ## Before and after The Limits step for τ³-bench, before and after: | Before | After | | --- | --- | | <img width="440" alt="Limits step on main: Scope only, with tasks, concurrency, attempts and two spending caps" src="https://github.com/user-attachments/assets/80b468a4-326f-4057-a4b2-5551ea93e642" /> | <img width="440" alt="Limits step on this branch: Scope with turns and model calls, Time limits in minutes, then a tau3-bench section with steps and tool errors" src="https://github.com/user-attachments/assets/6278494c-6c9f-456f-b739-9578494315ee" /> | ## Verified - Backend suite: 2175 tests OK. Route suite: 531 tests OK (after merging `main`, with #124's serving providers and #150's Boat VMs). - New tests: - A whole conversation that ends past the agent limit, though no single turn does. - Four one-call turns cut on the fourth call by `max_calls=3`. - The relay counts only answered calls on the agent's route. - A slow capability whose three adds each fit a 1 s limit while their total does not. - A capability run refuses `--max-calls`. - Recipe validation of the limits, including a value that rounds to 0 s. - τ³-bench steps above 200 are refused. - OSWorld's `--max_steps` is replaced, or added when the agent's command doesn't set it. - Against the installed τ³-bench upstream: the driver uses 200 steps and 10 errors by default, and 7 and 3 when the run sets them. - Checked in the browser, light and dark, for τ³-bench and OSWorld. ## Risk - **Agent timeouts on conversation benchmarks.** A conversation that used to fit 600 s per reply now has a single limit for the whole conversation, 7200 s on τ³-bench. A very long conversation can now time out where it didn't before. Undo by forcing a larger `--timeout agent=SECONDS`, or by reverting `limits.turn_seconds`. - **Comparability.** A run whose limits differ from the defaults is not yet marked as not comparable with the leaderboard. - **Overlap.** #174 (the agent's setup timed apart from its run) and #133 (a benchmark's own limits) change the same parts of `run.py` and the catalogue, so expect merge conflicts.
Ten commits landed on main since this branch left it; the unit loop and the trial writer conflicted. - standard/run.py, the unit loop: main runs the agent inside the unit's VM (unit_vm) and names the limits anew (agent_limit, phase_limit); this branch opens the task's door before the agent's phases and closes it after them. The VM, then the door, then the agent's setup and run, on main's limits. - standard/run.py, drive_agent: main dropped turn_limit for agent_limit, and starts the agent's clock once its session is open. The board's agent_run phase is emitted there with that same limit; the line this branch added still called turn_limit, which git merged without a conflict. - standard/run.py, unit(): over_acp() stays, with main's retry number. - standard/job.py, write_trial: both sides added parameters; all stay.
…ere it starts main lets a run force the agent's limit (--timeout agent). The board is told that limit for the agent's run, from its open session, and ACP's two start limits for its setup; the run command's page says an ACP agent starts before its limit does.
This comment has been minimized.
This comment has been minimized.
#173 merges first and words each phase's hover by the kind of run, a standard run's setup included; this branch's own wording of the same line would only conflict with it.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
aitianshi
added a commit
that referenced
this pull request
Nov 17, 2026
Closes #179 ## What A trial's page spent its first screen on parts that said nothing or said it twice, and hid some of what the trial recorded. This reworks it into a summary of the trial. - **Head.** The name is smaller. One row under it says what ran (the benchmark and its runner, or the importer of an imported trial; the agent with its mark, name, version and model) and how it went (the dataset's checks, the reward or the score, the agent's time, the steps). The Tool calls card and the evidence strip (Trajectory, Trace, Reward, Rate limits) are gone. A value the trial has not has no cell and no dash. - **Overview.** "Shape of the run" is gone. "Where the time went" is a compact tile in a grid that other tiles can join: its total in the heading, no paragraph, a chip for every phase the trial recorded however short, each with a hover saying what the phase holds on that kind of run. A panel with nothing to show is not drawn. - **Durations.** Two decimals wherever one is drawn: the `secs` filter (every page that uses it), the run page's script, a span's minutes (`nsdur`), `nullius runs show`. One decimal turned 0.05 s into "0.1s". - **Artifacts.** The tab lists only the records the trial wrote, each with Read and Download. A record's page has Copy and Download of the whole record, redacted as the reader shows it (`/trial/…/artifact/<key>/download`), and a back button to the Artifacts tab. A record past the reader's 16 MiB is saved as `<name>.partial.<ext>`. - **Tabs.** Artifacts, Conversation and the judge tabs no longer repeat their tab's name as a heading. The judge tabs keep the experimental badge (who answered, the question set, the date and the tokens in its hover) and say a refusal, an earlier vocabulary or a truncated state only when there is one. - **Removed.** The "benchmark's own judge" card, with the code that read `verifier/judge.json`: nothing in the application ever wrote that file. Its two hand-made fixture copies are deleted with it, which is why the judge's script floors this at severe. Commits nine and ten correct what an independent review of the first eight found: cells colliding in the head at mid widths, sideways scroll at phone width, an imported trial said to be Harbor's, dashes where the page claimed none, three places still at one decimal, a silent cut of large downloads, Copy saying copied when a browser refused the write, and tests whose names no longer matched what they asserted. The last three take `main` (ten commits, among them a task's attempts and an attempt's tries, each with tabs on this page): - The head reads: the name, what ran, then the attempt and try tabs, then that attempt's numbers. What is above the tabs is every attempt's, so a tab changes what is under it and the tabs stay put. A trial with no tab keeps its one row. - All attempts has the same head (one macro, `traces/_head.html`) in the place of its large name and subtitle. - Download now reads an earlier try's records, as the record page does; it answered 404 there. ## Touches - templates and static: `traces/trial.html`, `traces/_head.html` (new), `traces/attempts.html`, `_judgement.html`, `_conversation.html`, `artifact.html`, `runs/_execution.html` (its script's `secs`), `base.html` (the copy handler), `style.css`, `traces/conversation.css` - routes: `web/traces.py` (the download route, the importer's name), `web/filters.py` (`secs`, `nsdur`), `web/judge.py` - backend: `runs/artifacts.py` (`whole`), `runs/cli.py` (two decimals) - docs: `field-guide.md` (`n_rate_limited_retries`), `cli/pages.md` (the download route), the agent marks README - tests: route tests (`test_attempt_routes.py` among them), and a new `ui_tests/test_trial_head.py` ## Before and after `main` and this branch serving the same job trees, dark, 1280 px wide, rail collapsed. **Overview, a SWE-bench trial** | Before | After | | --- | --- | | <img width="420" alt="Before: cards, evidence strip, title, bar and paragraph" src="https://github.com/user-attachments/assets/1f59e693-1bb7-47a8-8407-3202eb96527a" /> | <img width="420" alt="After: one row of blocks and a time tile" src="https://github.com/user-attachments/assets/b9f34372-0954-4648-a885-84b647b9eb0d" /> | **Artifacts tab** | Before | After | | --- | --- | | <img width="420" alt="Before: nine rows, six of them not recorded" src="https://github.com/user-attachments/assets/f4b3f4fe-fe79-4248-90ec-c5a25b7bd534" /> | <img width="420" alt="After: the three records written, each with Read and Download" src="https://github.com/user-attachments/assets/7b1ffcab-ab3d-48b4-938f-cd1e2351f475" /> | **A record** | Before | After | | --- | --- | | <img width="420" alt="Before: a back link and the page" src="https://github.com/user-attachments/assets/c7207091-803e-4a45-9558-16d6242b30de" /> | <img width="420" alt="After: a back button, Copy and Download" src="https://github.com/user-attachments/assets/689dc2bd-d7cf-4f92-902f-c87b05945080" /> | **Judge tab** | Before | After | | --- | --- | | <img width="420" alt="Before: the benchmark judge card, a heading and a description" src="https://github.com/user-attachments/assets/ee428219-8582-4699-9847-d3f2cf11383a" /> | <img width="420" alt="After: the badge and the cards" src="https://github.com/user-attachments/assets/20365bea-f70d-4349-8552-af91fc62becf" /> | ## Verified - On the merge with `main` (`49d32d1`): backend suite 2229 OK, route suite 562 OK. With #174 merged on top as well: 2236 and 562 OK. - New tests: the head for a Harbor trial, a trial with nothing to show, a SWE-bench trial with and without its report, an imported trial, a scored trial; the cut download; the judge tab's notes; a span's minutes against `secs`; the run page's script against the template (run in node); a bar shorter than a second; the shell's phases against the page's. - Attempts: a retried attempt's earlier try downloads its own record (the test fails without the fix); the attempt and try tabs sit at the same height on every attempt of a task and on All attempts, at 820, 1024 and 1280 px, with no sideways scroll or clipped text at 390. - Browser, measured in the page: a six-cell SWE-bench trial at 390, 820, 1024 and 1280 px has no cell wider than its box and no sideways scroll; the Artifacts tab and a three-page record at 390 px fit; dark and light at 1280 px. - Copy: the whole record reaches the clipboard (22,990 characters of a record shown 8,000 a page) through each of its three paths, checked with the clipboard stubbed, and a real click reads "copied". - An independent review of the first eight commits: twelve findings, each fixed or answered in the last two commits. - Not verified: Safari (the Copy path written for it could not be run here), the judge tabs with a real judgement (a seeded record in a test), the nightly UI sweep. ## Risk - Every duration drawn through `secs` gains a second decimal, on every page that uses it. A stage time recorded to one decimal reads "41.20s". - The trial page no longer shows the count of retried rate limits; the job table's `429 ×N` badge keeps it. - Deleting the two `judge.json` fixtures removes the only trace of a card nothing could fill; the judge's script floors a deleted test file at severe. - Merges before #174, which no longer touches `traces/trial.html`: the two merge cleanly in that order. - Undone by a revert: no stored data changes shape.
🟡 PR judge · 8 of 8 yes · #174 ↔ #180
Acceptance criteria, 6 of 6 met
Checks the script runs itself, 4 of 4 pass
|
aitianshi
pushed a commit
that referenced
this pull request
Nov 19, 2026
…viders, set its own limits and prove a merged null control (#133) ## What A benchmark of the standard can now say more about how its units run and end, which a computer-use benchmark like OSWorld2 (#134) needs. `gated`, the new limits and `prove`'s new options are opt-in. The relay's rules and the rules of how a unit ends change for every benchmark of the standard; Risk lists them. - **Gated tasks** (`gated: true`): a unit's failure that the relay's records explain, a model endpoint's (the agent's, the judge's or the simulated user's), says in its `result.json` the status and class alone, never the provider's words, which can quote the request. The trial's call records keep the provider's words, and an error a driver raises itself is kept as the driver words it. - **A benchmark option's values by name** (`names`: {value: name}): the builder's cards and select, Review and the benchmark's page show the name, `about` stays its tooltip, and a recipe and the CLI keep the value. OSWorld1's agents are named as their upstream spells them. - **An option on its own step** (`step`): `where`, where the work runs, on the Where step, one value as fixed, and in Review's Where row; `dataset`, what some datasets need, on the Dataset step for those datasets. OSWorld1's VM provider is on Where. - **Limits of a benchmark's own:** - `limits.score_seconds`: how long `score` may take, the benchmark's level below `--timeout score=` (#191); the Limits step shows it as the benchmark's; - `limits.call_seconds`: how long the relay waits for one model call, also `relay serve --timeout`; - **How a unit ends:** - a driver's error that begins `EnvironmentSetupError: `, `VerifierError: `, `CancelledError: ` or `CauseUnknownError: ` keeps that kind; - the agent's own last call refused as longer than the model's context window is `ContextWindowExceededError`, the agent's; - a last 429 is read from the agent's own calls in the order they began, so a side model's later 200 does not cover it; - a unit a guard's stop ended keeps the score its driver gave, its own last call's 429 notwithstanding, unless that 429 said the credit is spent. `score` can say the stop's reason (`verifier_result.stopped`); - with #154's `--retries`, a unit a guard asked to stop and a full context window are not tried again; - for an upstream's own answerer: `OutputLengthExceededError` only when its last reply has neither text nor a tool call. When the answerer fails and its last reply was a Responses object answered 200 and `failed`, that is an API error. - **The relay:** - a non-streamed 200 to a model call with no reply in it reaches the caller and the record as a 502, or the 429 or 402 of an error it held (the record keeps `answered: 200`). #124 leaves counting an error inside a 200 to a pull request of its own; this one takes only a 200 with no reply in it; - a stream cut off midway, after the caller had its 200, is recorded as a 502, and its booking against the spending cap stays whole; - a call refused with 400, 413 or 422 records each inline image it sent by its declared type, the type its bytes show and its size, never a byte; - a `stop` file beside the relay's records: the relay notes in it an agent's call refused for the model's context window, and that note refuses nothing. Once a benchmark's guard writes a line of its own there, the agent's later calls are refused unsent, while side models and the judge's and simulated user's routes go on; - a failed Responses object with no usage records a cost of 0, and its booking is kept. - **A job's name:** - a run holds its job name until its process ends, and one under a name another run still holds is refused before it writes anything; - a rerun into the same job moves each rerun trial's earlier records (`.work/relay`, `result.json`, `verifier/`, `agent/`) aside to `.work/earlier.<ns>`. - **prove and merge:** - `nullius standard prove ... --bench-option NAME=VALUE` sets the benchmark's own options for the units check and the controls; - `--job nop=MERGED` (or `oracle=`) checks a merged launch of a control against its bound instead of running it, and every trial's own record; - `merge` accepts shards that differ in benchmark options and keeps each shard's in `merged_from`; - `merge` refuses, unless `--force`, a trial whose record names another job, benchmark, version, control or agent, and a job itself merged despite differences. It lists the shards absent from a merge only when the launch has at most 1,000. - **Smaller:** - a trace that cannot be built leaves `agent/trace.error.txt`; - an Original's trajectory keeps its tool calls; - each call's record keeps the request's sha256; - `thinking_tokens` count as reasoning; - a setup's `comparable` can list the models whose runs compare with the paper, and the builder's message for another model says so; - a test checks that every file a host's snapshot takes of a bundled entry is package data. **Read first:** `failure`, then `_hollow` and the stop in `Relay._forward` (`app/nullius/standard/relay.py`). Next, the outcome rules in `Run` (`app/nullius/standard/run.py`), then `names` and `step` in `recipe.compile` and `benchOptions` in the builder, the `--job` check in `prove.py` and the trial check in `merge.py`. Then `test_standard_gated.py`, `test_standard_context_window.py` and `test_standard_controls.py`. The rest is the definitions' checks, the CLI options, the sidecar's argument and the pages. ## Touches the standard (`app/nullius/standard/`: `relay`, `run`, `prove`, `definitions`, `merge`, `job`, `cli`, `container`, `recipe`, `fidelity`, `retries`) · the builder (`app/static/builder/`, `app/web/builder.py`: the score's default on the Limits step, an option's names, and an option on its own step) · the benchmark page (an option's names) · OSWorld1's definition (its agents' names, its VM provider on Where) · docs and site (`docs/standard.md`, `docs/cli/run.md`, `docs/cli/read-jobs.md`, `docs/builder.md`, `docs/settings/model-providers.md`) ## Verified - At `e768c632`, on main at `db6e8f96`, on Python 3.12: backend suite (`python3 app/tests/parallel.py`), 2310 tests in 129 modules, OK; route suite with Node on PATH, 639 tests in 53 modules, OK; the wheel's check (`app/packaging/verify.py`), OK. - **An option's names and its step, in a headless Chromium, light and dark, 1400 and 390 px,** with #134 on top: OSWorld1's agents by name in their select, OSWorld2's in their cards; the VM provider of each as a fixed value on the Where step, shown only while a participant that takes it is chosen; OSWorld2's Website host suffix on the Dataset step for All tasks, not for Offline, no services; Served by empty on any choice of model. No page error. - **The 12 changed test modules on main's source** (`5bcfd82d`): 9 fail, 2 pass, and 1 does not end. - `test_standard_context_window`: 19 of 20; `test_standard_controls`: 17 of 43; `test_standard_relay`: 8 of 58; `test_standard_gated`: 4 of 6. - `test_standard_run`: 2 of 50; `test_standard_trace`: 2 of 20; `test_standard_parts`, `test_standard_served_by` and `test_standard_units`: 1 each. - `test_standard_shard` does not end within 4 minutes: one of its merges names 2^64 shards, and main's `merge` lists every absent one. - `test_standard_versions` and `test_dependency_surface` pass on both. - **Covered in #134, not here:** that `limits.call_seconds` reaches the relay and the sidecar's `--timeout`. OSWorld2 is the first entry to set it (`test_osworld2_muse.py`). - **#124's own tests.** Three were changed for two deliberate differences: a 200 with no reply is a 502, and a failed Responses object is read as failed. The rest of `test_standard_served_by` passes as main has it; `write_shard` in `test_standard_shard` takes #137's `reward` beside this branch's `bench_options`. - `PYTHONPATH=app python3 -m nullius standard check`: ok. `python3 app/tests/catalogue_docs.py --write`, then the check: up to date. `python3 app/tests/field_contract.py`: ok. `app/site/readme_media.py --check`: current. The docs check reports the same two findings in `AGENTS.md` as on main. - Not run: a benchmark already on main, live, on this branch; the container backend with `--timeout`; the docker tier; the site build. ## Risk - **For every benchmark of the standard:** - a non-streamed 200 with no reply is a 502, and a stream cut off midway is recorded as one; - the agent's own last call refused for its context window is the agent's error (`ContextWindowExceededError`), where it was an API error outside its rate; - an agent's last 429 followed by a side model's 200 makes the unit `rate_limited`, where its score stood; - an upstream answerer whose last reply was a `failed` Responses object is an API error, where it was the agent's; - a driver's error of a kind it names: a setup error or a stop is left out of the agent's rate, and a verifier error or one of no known cause counts as unknown. A wrong kind would move a unit between them; - `prove` runs no control when the units check failed, and fails a control job with a unit that has neither a score nor an error; - a run under a job name another run still holds is refused, and a rerun into an existing job moves the rerun trials' records aside; - `merge` refuses a trial whose record names another job unless `--force`. - **Records.** Stored records gain fields and keep the old ones. - These commits sit on main at `db6e8f96`, with #124, #150, #191, #154, #174 and the Recipes page (#205). #124's `--served-by` code is main's as merged; #191's time limits come first, so `--timeout score=` is used over `limits.score_seconds`, and a unit's cap on model calls (`UnitCallCapError`) is read before this PR's outcome rules. #154's `--retries` replaces the one more run of a rate-limited unit, so the earlier `limits.run_again: false` that switched it off is gone; main counts a provider's 402 as a usage limit, and the tests follow. The first and fourth commits gave a definition pins of its serving providers, defaults of Served by after review; the last commit removes them, as the reviewer asked, since no benchmark declares one: the diff over main has none, and Served by is the user's alone, as in #124. - The fifth commit shortens this PR's docs and messages. No logic changes. - **Names and Where.** An option's names change only what is shown; a recipe saved before keeps working. An option of where the work runs leaves the Agent step for Where, and one some datasets need is asked on Dataset: OSWorld1's VM provider is now on Where. - Undone by reverting the commits. --------- Co-authored-by: Oleg <247799477+sirotaoleg@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #180
What
A standard trial of an agent driven over ACP had no agent setup: the agent's process start,
initializeandsession/new(which also starts the benchmark's MCP servers) ran inside the window written asagent_execution. The trial page read "Setup —" and its Agent time carried the start.The run now notes when the agent's session opens and writes Harbor's
agent_setup, from the end of the environment's setup to the open session.agent_execution, and the window of the trial's trace, start there.The second commit corrects what an independent review of the first found:
agent_execution, as Harbor writes a failed agent setup.agent_setupphase, so the live board disagreed with the trial page. It emits the phase, starts the agent's limit when the session opens, and takes every duration from the trial's result.The last two commits take
main(ten commits: a unit's VM, forced time limits, attempts and retries among them):mainrenamed the turn's limit to the agent's whole work on a unit (agent_limit) and starts its clock once the session is open. The board'sagent_runphase is emitted there with that same limit, a run's--timeout agentincluded; the setup keeps ACP's two start limits. The line this branch had added still called the old name and merged without a conflict: every ACP unit would have ended in an error.Installing the agent stays a stage of the run, not of a trial.
Touches
standard/run.py(the unit loop,drive_agent),standard/job.py(write_trial,feed_durations),standard/acp.py(its two start limits, named)field-guide.md(the phase rows, shorter)test_standard_trace.py,test_standard_run.pyBefore and after
The same recipe (
agent-task-v1, the reference agent, a stand-in endpoint answering in 2 s) run on #173, which merges first, and on #173 with this branch: the trial page this lands on. Before, the agent's start sits in Agent; after, it is Setup.Verified
main(49d32d1): backend suite 2236 OK, route suite 547 OK. With feat(traces): rework the trial page around what a trial recorded #173 merged first: 2236 and 562 OK.write_trial. The trace's window is pinned as well: it starts at the open session, and a model call made during the setup widens it and is never clamped.--timeout agentand reads the feed: the setup carries ACP's 300 s, the run the forced limit. With the old limit's name left in, the setup test ends in an error.nullius standard run agent-task-v1 reference --limit 1 --unsandboxed --timeout agent=45, stand-in endpoint):result.jsonholdsenvironment_setup0.046 s,agent_setup0.054 s,agent_execution2.056 s,verifier0.001 s, back to back;events.jsonlholds the phasesenv_setup(600 s),agent_setup(300 s),agent_run(45 s),verify(600 s) and atrial.endwith the same four durations; the trace's session is 2.056 s.Risk
agent_execution. Readers already meet a trial without one (a unit that fails in its environment's setup).