Repository navigation
feat(standard): force each phase's time limit, bound an agent's whole work on a unit and cap its model calls - #191
Merged
Conversation
… work on a unit and cap its model calls A run can force every phase's time limit (--timeout PHASE=SECONDS), the turns of a conversation (--max-turns) and the model calls an agent makes on a unit (--max-calls, refused by the relay as the agent's error). The agent's limit now covers its whole work on a unit, every turn and the simulated user's between them; a capability keeps a limit per add and recall. tau3-bench's conversation limit is 7200 s. A benchmark's own bounds are options marked `limit`: tau3-bench's steps (1 to 200) and tool errors per conversation, OSWorld's steps per task, each agent's own by default (`by`). The builder's Limits step shows Scope, Time limits in minutes, then a section named after the benchmark; choosing Original presets its agent limit. Closes #190
…cability # Conflicts: # app/nullius/standard/cli.py # app/nullius/standard/container.py # app/nullius/standard/job.py # app/nullius/standard/recipe.py # app/nullius/standard/relay.py # app/nullius/standard/run.py # app/static/builder/command-builder-core.js # docs/builder.md # docs/cli/run.md # docs/standard.md
This comment has been minimized.
This comment has been minimized.
…cability # Conflicts: # app/nullius/standard/cli.py # app/nullius/standard/recipe.py # app/nullius/standard/run.py # docs/builder.md # docs/cli/run.md
This comment has been minimized.
This comment has been minimized.
…s value, Nullius's default greyed An empty limit field shows the value the run uses. The benchmark's own (its limits, a setup's own as its Original declares it, each task's) and the agent's own read in the text colour, as a value would; Nullius's default stays a grey placeholder. Priority, low to high: Nullius, agent, benchmark. Choosing Original no longer types its agent limit into the field, which forced it on every participant over each task's own.
🔴 PR judge · 7 of 8 yes · #191 ↔ #190
Acceptance criteria, 4 of 9 met
Checks the script runs itself, 4 of 4 pass
some of the diff was cut before the judge read it |
aitianshi
pushed a commit
that referenced
this pull request
Nov 19, 2026
…viders, set its own limits and prove a merged null control (#133) ## What A benchmark of the standard can now say more about how its units run and end, which a computer-use benchmark like OSWorld2 (#134) needs. `gated`, the new limits and `prove`'s new options are opt-in. The relay's rules and the rules of how a unit ends change for every benchmark of the standard; Risk lists them. - **Gated tasks** (`gated: true`): a unit's failure that the relay's records explain, a model endpoint's (the agent's, the judge's or the simulated user's), says in its `result.json` the status and class alone, never the provider's words, which can quote the request. The trial's call records keep the provider's words, and an error a driver raises itself is kept as the driver words it. - **A benchmark option's values by name** (`names`: {value: name}): the builder's cards and select, Review and the benchmark's page show the name, `about` stays its tooltip, and a recipe and the CLI keep the value. OSWorld1's agents are named as their upstream spells them. - **An option on its own step** (`step`): `where`, where the work runs, on the Where step, one value as fixed, and in Review's Where row; `dataset`, what some datasets need, on the Dataset step for those datasets. OSWorld1's VM provider is on Where. - **Limits of a benchmark's own:** - `limits.score_seconds`: how long `score` may take, the benchmark's level below `--timeout score=` (#191); the Limits step shows it as the benchmark's; - `limits.call_seconds`: how long the relay waits for one model call, also `relay serve --timeout`; - **How a unit ends:** - a driver's error that begins `EnvironmentSetupError: `, `VerifierError: `, `CancelledError: ` or `CauseUnknownError: ` keeps that kind; - the agent's own last call refused as longer than the model's context window is `ContextWindowExceededError`, the agent's; - a last 429 is read from the agent's own calls in the order they began, so a side model's later 200 does not cover it; - a unit a guard's stop ended keeps the score its driver gave, its own last call's 429 notwithstanding, unless that 429 said the credit is spent. `score` can say the stop's reason (`verifier_result.stopped`); - with #154's `--retries`, a unit a guard asked to stop and a full context window are not tried again; - for an upstream's own answerer: `OutputLengthExceededError` only when its last reply has neither text nor a tool call. When the answerer fails and its last reply was a Responses object answered 200 and `failed`, that is an API error. - **The relay:** - a non-streamed 200 to a model call with no reply in it reaches the caller and the record as a 502, or the 429 or 402 of an error it held (the record keeps `answered: 200`). #124 leaves counting an error inside a 200 to a pull request of its own; this one takes only a 200 with no reply in it; - a stream cut off midway, after the caller had its 200, is recorded as a 502, and its booking against the spending cap stays whole; - a call refused with 400, 413 or 422 records each inline image it sent by its declared type, the type its bytes show and its size, never a byte; - a `stop` file beside the relay's records: the relay notes in it an agent's call refused for the model's context window, and that note refuses nothing. Once a benchmark's guard writes a line of its own there, the agent's later calls are refused unsent, while side models and the judge's and simulated user's routes go on; - a failed Responses object with no usage records a cost of 0, and its booking is kept. - **A job's name:** - a run holds its job name until its process ends, and one under a name another run still holds is refused before it writes anything; - a rerun into the same job moves each rerun trial's earlier records (`.work/relay`, `result.json`, `verifier/`, `agent/`) aside to `.work/earlier.<ns>`. - **prove and merge:** - `nullius standard prove ... --bench-option NAME=VALUE` sets the benchmark's own options for the units check and the controls; - `--job nop=MERGED` (or `oracle=`) checks a merged launch of a control against its bound instead of running it, and every trial's own record; - `merge` accepts shards that differ in benchmark options and keeps each shard's in `merged_from`; - `merge` refuses, unless `--force`, a trial whose record names another job, benchmark, version, control or agent, and a job itself merged despite differences. It lists the shards absent from a merge only when the launch has at most 1,000. - **Smaller:** - a trace that cannot be built leaves `agent/trace.error.txt`; - an Original's trajectory keeps its tool calls; - each call's record keeps the request's sha256; - `thinking_tokens` count as reasoning; - a setup's `comparable` can list the models whose runs compare with the paper, and the builder's message for another model says so; - a test checks that every file a host's snapshot takes of a bundled entry is package data. **Read first:** `failure`, then `_hollow` and the stop in `Relay._forward` (`app/nullius/standard/relay.py`). Next, the outcome rules in `Run` (`app/nullius/standard/run.py`), then `names` and `step` in `recipe.compile` and `benchOptions` in the builder, the `--job` check in `prove.py` and the trial check in `merge.py`. Then `test_standard_gated.py`, `test_standard_context_window.py` and `test_standard_controls.py`. The rest is the definitions' checks, the CLI options, the sidecar's argument and the pages. ## Touches the standard (`app/nullius/standard/`: `relay`, `run`, `prove`, `definitions`, `merge`, `job`, `cli`, `container`, `recipe`, `fidelity`, `retries`) · the builder (`app/static/builder/`, `app/web/builder.py`: the score's default on the Limits step, an option's names, and an option on its own step) · the benchmark page (an option's names) · OSWorld1's definition (its agents' names, its VM provider on Where) · docs and site (`docs/standard.md`, `docs/cli/run.md`, `docs/cli/read-jobs.md`, `docs/builder.md`, `docs/settings/model-providers.md`) ## Verified - At `e768c632`, on main at `db6e8f96`, on Python 3.12: backend suite (`python3 app/tests/parallel.py`), 2310 tests in 129 modules, OK; route suite with Node on PATH, 639 tests in 53 modules, OK; the wheel's check (`app/packaging/verify.py`), OK. - **An option's names and its step, in a headless Chromium, light and dark, 1400 and 390 px,** with #134 on top: OSWorld1's agents by name in their select, OSWorld2's in their cards; the VM provider of each as a fixed value on the Where step, shown only while a participant that takes it is chosen; OSWorld2's Website host suffix on the Dataset step for All tasks, not for Offline, no services; Served by empty on any choice of model. No page error. - **The 12 changed test modules on main's source** (`5bcfd82d`): 9 fail, 2 pass, and 1 does not end. - `test_standard_context_window`: 19 of 20; `test_standard_controls`: 17 of 43; `test_standard_relay`: 8 of 58; `test_standard_gated`: 4 of 6. - `test_standard_run`: 2 of 50; `test_standard_trace`: 2 of 20; `test_standard_parts`, `test_standard_served_by` and `test_standard_units`: 1 each. - `test_standard_shard` does not end within 4 minutes: one of its merges names 2^64 shards, and main's `merge` lists every absent one. - `test_standard_versions` and `test_dependency_surface` pass on both. - **Covered in #134, not here:** that `limits.call_seconds` reaches the relay and the sidecar's `--timeout`. OSWorld2 is the first entry to set it (`test_osworld2_muse.py`). - **#124's own tests.** Three were changed for two deliberate differences: a 200 with no reply is a 502, and a failed Responses object is read as failed. The rest of `test_standard_served_by` passes as main has it; `write_shard` in `test_standard_shard` takes #137's `reward` beside this branch's `bench_options`. - `PYTHONPATH=app python3 -m nullius standard check`: ok. `python3 app/tests/catalogue_docs.py --write`, then the check: up to date. `python3 app/tests/field_contract.py`: ok. `app/site/readme_media.py --check`: current. The docs check reports the same two findings in `AGENTS.md` as on main. - Not run: a benchmark already on main, live, on this branch; the container backend with `--timeout`; the docker tier; the site build. ## Risk - **For every benchmark of the standard:** - a non-streamed 200 with no reply is a 502, and a stream cut off midway is recorded as one; - the agent's own last call refused for its context window is the agent's error (`ContextWindowExceededError`), where it was an API error outside its rate; - an agent's last 429 followed by a side model's 200 makes the unit `rate_limited`, where its score stood; - an upstream answerer whose last reply was a `failed` Responses object is an API error, where it was the agent's; - a driver's error of a kind it names: a setup error or a stop is left out of the agent's rate, and a verifier error or one of no known cause counts as unknown. A wrong kind would move a unit between them; - `prove` runs no control when the units check failed, and fails a control job with a unit that has neither a score nor an error; - a run under a job name another run still holds is refused, and a rerun into an existing job moves the rerun trials' records aside; - `merge` refuses a trial whose record names another job unless `--force`. - **Records.** Stored records gain fields and keep the old ones. - These commits sit on main at `db6e8f96`, with #124, #150, #191, #154, #174 and the Recipes page (#205). #124's `--served-by` code is main's as merged; #191's time limits come first, so `--timeout score=` is used over `limits.score_seconds`, and a unit's cap on model calls (`UnitCallCapError`) is read before this PR's outcome rules. #154's `--retries` replaces the one more run of a rate-limited unit, so the earlier `limits.run_again: false` that switched it off is gone; main counts a provider's 402 as a usage limit, and the tests follow. The first and fourth commits gave a definition pins of its serving providers, defaults of Served by after review; the last commit removes them, as the reviewer asked, since no benchmark declares one: the diff over main has none, and Served by is the user's alone, as in #124. - The fifth commit shortens this PR's docs and messages. No logic changes. - **Names and Where.** An option's names change only what is shown; a recipe saved before keeps working. An option of where the work runs leaves the Agent step for Where, and one some datasets need is asked on Dataset: OSWorld1's VM provider is now on Where. - Undone by reverting the commits. --------- Co-authored-by: Oleg <247799477+sirotaoleg@users.noreply.github.com>
aitianshi
added a commit
that referenced
this pull request
Nov 19, 2026
…ces on the machine and a null control (#134) Stacked on #133: this pull request's own commits are the last sixteen. Until #133 is merged, the diff shows its files too. It can be merged after #133. ## What OSWorld-V2's 108 tasks become a bundled benchmark, `osworld2`, on the Nullius standard, at the upstream's release `osworld-v2.1`. Each task is a unit in an Ubuntu VM of its own on the upstream's docker provider, set up by the task's own class and scored from 0 to 1 by its own evaluate(). The tasks and their assets are gated on Hugging Face. Nullius does not fetch them, and the entry is `gated` (#133): a unit's failure from a model endpoint says its status alone, and the driver names an upstream exception by a known class's name at most, never by its words. The Original setup runs one of OSWorld-V2's own agents, M3 or Muse Spark, chosen by the Agent option, from its published script, with every model call sent to the relay. Claude Sonnet 4.6 is the LLM grader of 21 tasks and the simulated user of 6. No provider is pinned: Served by is the user's, as for OSWorld1. Only M3's runs count as comparable with the paper's row. Its Review row and its generated page name the setup "OSWorld-V2 agents". A null control runs one `DONE` action, then the task's evaluate() on the untouched VM. The entry declares `agent.computer/v1`. No agent from the Agents page runs it yet. Its texts are one Good to know line and 9 one-line limitations. The mock websites of 40 tasks are OSWorld-web at the same tag. The guide's steps build them from source and serve them on the machine the units run on. A unit that needs them is refused before its VM starts when they do not answer. For 026 and 041, Nullius makes GitLab on the run's machine before the first of them, with a root token it keeps there, then starts it anew from its snapshot for each, one task at a time; Overleaf's 2 tasks get Overleaf started anew. Each of the 8 tasks that ask for a proxy gets a proxy of its own. The options are the agent, by name, Thinking for M3, and disk write limits for each VM. The website host suffix is asked on the Dataset step, for the four datasets with website or GitLab tasks. Steps per task is with the limits, each agent's own by default, as its wrapper runs it: 100 for M3, 500 for Muse Spark. The VM provider, Docker alone, is on the Where step. By default M3 takes 100 steps with thinking off, as the upstream's M3 script does; the paper ran M3 at 500 steps with thinking on. Muse Spark runs as published at this release, while its published score came from newer upstream code; a limitation says it does not compare, and a docs page says what differs. There are six datasets: All tasks (108, the upstream's `test_v2.json`), Offline (82), and four by the services their tasks need. Tasks 103 and 104 are not supported: their evaluate() needs FreeCAD on the driver's machine. Review and the run's start ask for Docker and for the scoring programs (ImageMagick, ffmpeg, tesseract) on the driver's machine. As for OSWorld1, the Where step greys out the container card under the Original, whose docker provider needs Docker on the machine. A guide, `docs/benchmarks/osworld2-machine.md`, prepares a machine, runs the null control on one task as its last setup step, and says what a run takes in time and money. A second page, `docs/benchmarks/osworld2-results.md`, says how a run differs from the paper and the upstream, and how a unit is scored, stopped or left unscored. ## Touches standard (an option's example and a machine setup's label: the definition check, the benchmark and agent pages, the builder, Review) · catalogue · templates and static (the mark) · docs and site (`docs/benchmarks/`, the generated pages, `AGENTS.md`) · routes (tests only) · packaging (the entry's `.yml`, `.Dockerfile` and `startup/*.py` as package data in `pyproject.toml`) ## Before and after **Before:** The Benchmarks page before this change  **After:** OSWorld2 listed, with its mark  **Before:** The builder's Benchmark step under Computer use: OSWorld1, beside the selected GAIA  **After:** OSWorld1 and OSWorld2  **Before:** The Agents page: "n/7 benchmarks it runs"  **After:** "n/8": OSWorld2 counts, and no agent runs it  **Before:** An agent's page, Codex: the benchmarks it runs  **After:** OSWorld2 listed: it needs `agent.computer/v1`, which Codex does not offer  **New:** OSWorld2's page: its setup and its proofs by dataset  **New:** Its options: the agent by name, steps, thinking, the VM provider, the website host suffix and the disk write limits  **New:** Its dataset and its Good to know line  **New:** Its limitations  **New:** Dataset step: six datasets with their controls; All tasks chosen asks for the website host suffix, and for no token  **New:** Agent step: OSWorld-V2 agents; the Agents page's agents lack computer use; the settings every agent shares, the disk write limits  **New:** The Original's own settings: MiniMax M3 or Muse Spark by name, and thinking  **New:** Muse Spark chosen: Thinking, an M3 setting, is gone  **New:** Model step, Like the paper: MiniMax-M3 on OpenRouter, Claude Sonnet 4.6 as the judge, each served by any provider until one is named  **New:** Limits step: Steps per task, each agent's own by default, 100 for MiniMax M3  **New:** Where step: the VM provider, Docker alone; no container or Boat backend here; Docker and the scoring programs checked on this machine  **New:** Review: every decision, both prerequisites ready, no model key in this fresh profile  **New:** The null control's merged job on test_v2, its first rows  **New:** 052 by the test stand's model in M3: 1.0  Taken at 1400 px, light theme, on a fresh profile. Cut to part of their page: the Benchmark step and Agents pairs (the top 1000 px), Codex's page (the top 1100 px), OSWorld2's page (four parts) and the null control's job (its first rows). The two jobs are runs named under Verified, opened in the app. The Model step, Review, OSWorld2's options, dataset and limitations, and the Dataset, Agent and Where steps were taken again after main's Served by (#124), Boat (#150) and time limits (#191), the shorter texts and GitLab made by Nullius, and checked at `273cca5e`: the other pictures differ there only in the sidebar, which they leave out. The two jobs were taken again at `c4427915`, on main at `0d83e393`, whose job page names the judge's button anew (#196); the other pages and steps pictured are the same there. OSWorld2's options and limitations and the Dataset, Agent, Model, Limits, Where and Review steps were taken again at `9526a5b1`, after the agents' names, the steps as a limit, the VM provider on Where, the website host suffix on Dataset and Served by with no default. ## Verified - At `9526a5b1`, on #133 (`e768c632`) on main at `db6e8f96`, on Python 3.12: backend suite (`python3 app/tests/parallel.py`), 2688 tests in 144 modules, OK; route suite with Node on PATH, 643 tests in 53 modules, OK; the wheel's check (`app/packaging/verify.py`), OK. - **OSWorld2's 15 test modules,** 375 tests, OK; they need the entry, so none passes without it. 10 are skipped here: they run only when the environment names the upstream with its gated task classes, OSWorld-web or GitLab. Three route test modules and three backend test modules changed for the new entry: the benchmark lists and counts, the Where step's container card, Settings' custom keys, and OSWorld2's case of gated tasks (`test_standard_gated.py`). Option examples and setup labels change `test_standard_options.py`, `test_prerequisites.py` and `test_standard_bench_options.py`, and the route modules `test_catalogue_routes.py`, `test_agents.py` and `test_builder_where.py`. - At `9526a5b1`, in a headless Chromium, light and dark, 1400 and 390 px: the builder's Agent step (the agents by name, Steps no longer there), Model step (Any of your choice: no provider filled in), Limits step (Steps per task, 100 for M3 and 500 for Muse Spark), Where step (VM provider docker, fixed) and Review; no page error. - In a headless Chromium, light and dark, 1400 and 390 px: the Benchmarks page, OSWorld2's page, the Agents page, Settings and the builder, no page error or cut text. OSWorld2's limitations fit one screen at 390 px, light and dark. `/ui-sweep --only osworld2` at `fc0bdb1f`, this pull request's third commit before the rebases, dropped since (it set `limits.run_again`, which main's `--retries` replaced): 6 screenshots, none with a page error or cut text. - `PYTHONPATH=app python3 -m nullius standard check`: ok. `python3 app/tests/catalogue_docs.py --write`, then the check: up to date. `python3 app/tests/field_contract.py`: ok. `app/site/readme_media.py --check`: current. The docs check reports the same two findings in `AGENTS.md` as on main. - **Where the runs below ran.** The commits named here are of the port's working branches, not in this repository: - `c3637ffd` is this pull request's tree with main's two later commits merged in; - `f74d1a00` and `58c2b586` add one test commit: two JSON files, no code. It gives OSWorld2's Original one more model, MiniMax M3.1 Flash, the name it is served as, its pin to MiniMax and `--max_tokens 32768`. That model is a test stand, not in this pull request; - `58c2b586` is this entry on the core before #124; - `34e35370` and `8e2e381b` are of the port before its move to this repository's history. - **The controls,** recorded in `standard.json`'s `proven`: - Offline, no services, on 2026-11-04, on an earlier commit: the 36 ids are the same twice, and the null control scores 0 on 4 tasks; - All tasks, on 2026-11-15, on `fc0bdb1f`, this pull request's third commit before the rebases, dropped since (it set `limits.run_again`, which main's `--retries` replaced); the next commit only writes this record, and the one after changes three tests. The null control ran on all 108 tasks, one launch with each task a shard of its own, at 16 MB/s and 160 IOPS a VM, in six lanes with at most seven VMs at a time; it took 1 h 32 min. 108 and 093 failed their setup under that load and passed alone at 24 MB/s and 240 IOPS, in 24 more minutes. The grader cost $0.14. `nullius standard prove osworld2 --dataset test_v2 --job nop=<merged job>` passed: the 108 ids are the same twice, 0 of 108 at full score, a mean of 0.0021 against a bound of at most 0.01. Five tasks score above 0 on the untouched VM, as a limitation says; - every task is in line with the same control on all 108 tasks on `34e35370`, which had the same mean; 077's LLM grader gave 0.054 both times. - **GitLab made by Nullius,** on a server whose GitLab had been set up by hand: the null control of 026, alone, made GitLab first (cloned, configured, its root token, its runner, its snapshot) and scored 0.0 in 15 min; 041, alone, started from that snapshot and scored 0.0 in 7 min; a snapshot of an older stamp was made anew by itself. No model call, no GitLab left after either, and the token in no file of the jobs. - **An agent's score on every kind of task,** each above the same task's null control (0.0 in each): - **Seven kinds,** on `58c2b586`, by the test stand's model in OSWorld-V2's M3 agent, from the command line: - desktop apps, 045: 1.0; - mock websites, 052: 1.0; - Overleaf, 072: 0.775; - GitLab, 026: 0.5, a partial score; - a proxy, 075: 1.0 (recorded 0.9999999999999999), with the unit's proxy carrying new outbound connections while the agent acted; - the public internet, 027: 1.0; - the LLM grader, 003: 1.0, with both grader calls answered 200. Each of the seven gives its recorded score again at `c3637ffd` when its outcome is computed from its saved records, with no VM and no evaluate(). - **The eighth, the simulated user:** 007, 0.8, by Muse Spark on `8e2e381b`; the simulated user answered 7 times. It was not run on this base. - **Live on this base, against OpenRouter:** - on `c3637ffd`, the null control on 087, a graded task: 0.0. The grader's call went to Anthropic by the definition's pin, and `standard.served` reads `held: true`; - on `f74d1a00`, the test stand's model on 052: 1.0. All 68 calls went to MiniMax by the test commit's pin and named it, `held: true`, with no error. - **This pull request's own agents** have no positive score on this base. M3 ran one task here, 017, on the core before #124 (`90fde73e`), and scored 0.0: the agent declared the task failed at its 120th step. Muse Spark ran none. Their scores before the move, on `34e35370` and earlier: - M3 on 9 tasks: 070 1.0 twice, 030 1.0, 045 1.0, 013 1.0 then 0.0, 031 0.714, 017 0.429, 010 0.0 three times and once an environment error; 026 and 098 left no score (a full context window, a spend guard); - Muse Spark on 3 tasks: 045 1.0, 007 0.8; 052 left no score (a launch error of ours, then a spend guard). - **Every attempt before the move,** two test stands' models included, was recorded and triaged: 61 attempts on 33 tasks. 43 were scored (21 full, 11 partial, 11 zero), 10 ended in an environment error and 8 were stopped by a guard or by hand. None was an agent error. Each of their full and partial scores was checked for the false-success patterns the port's audits found: none. - Not run: a positive score of M3 or Muse Spark on this base; the simulated user on this base; task 041 with an agent; a parity run of the paper's 108 tasks; the 10 skipped tests with the upstream named; the docker tier; the site build. ## Risk - OSWorld2's driver and OSWorld-V2's scripts run only when someone runs OSWorld2. The Agents page counts eight agent benchmarks, and the generated agent pages gain an OSWorld2 column. No agent there declares `agent.computer/v1`. - **Scores.** A run compares with the paper's row by its model alone: Docker instead of AWS, no US residential proxy, release `osworld-v2.1`, the grader through OpenRouter, and by default 100 steps with thinking off. A wrong error kind would move a unit between "environment error" (left out of the adjusted rate) and the agent's error (counted). - **Steps.** A Muse Spark run that sets no step limit now takes up to 500 steps, its wrapper's budget, where it took 100: a longer and dearer episode. M3 keeps 100. A saved recipe or command that names a number keeps it. - **The machine.** - VMs, the mock websites, GitLab and the proxies run on the machine the units run on, outside a run's constraints in Settings; `-j` bounds them. - GitLab's Docker in Docker and the websites' image build run as privileged containers. - In each of the driver's processes, `startup/sitecustomize.py` wraps the creation of the VM's container: its network, its ports on 127.0.0.1, its disk and its write limits. The VMs need a free Docker subnet. - The VM's clock is UTC+8: a dated task set up before 16:00 UTC and acted on after can score 0. - The first unit of 026 or 041 on a machine clones Task-Web/gitlab at its pin, pulls its images (about 7 GB) and makes its snapshot: about 8 minutes more, once. - **Secrets.** GitLab's root token is made on the run's machine and kept there in a file for its user alone, beside GitLab's clone; only the processes of 026 and 041 get it. Model keys stay with the relay. Tasks 001, 006 and 010 start their mail client offline, unlike the upstream. - Stacked on #133. Once it is merged, this branch is rebased onto main. - Undone by reverting the commits; the catalogue entry goes with them. --------- Co-authored-by: Oleg <247799477+sirotaoleg@users.noreply.github.com> Co-authored-by: Etienne Filhol <etienne.filhol@hotmail.fr>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #190
What
A standard run can now force its limits, and the agent's time limit covers its whole work on a unit.
run --timeout PHASE=SECONDSforces the limit of a phase:setup,image,agent,verifyorscore. Empty keeps the default, which comes from the task, then the benchmark, then Nullius.config.jsonrecords what was forced understandard.timeouts.limits.turn_secondsgoes from 600 s a reply to 7200 s a conversation.run --max-turns Nsets the turns of a conversation. The builder offers it only on a benchmark that holds conversations.run --max-calls Ncaps the model calls an agent makes on a unit. The relay counts each answered call on the agent's route; rate-limited and failed calls are not counted, and neither are the judge's or the simulator's. Past the cap the relay refuses the call with a 402, and the unit ends asUnitCallCapError, an agent error that is not scored. It is for agents only: a capability's route also carries the benchmark's reader, so a capability run refuses it.limit. τ³-bench gets Steps per conversation (--max-steps, from 1 up to its default of 200) and Tool errors per conversation (--max-errors). OSWorld gets Steps per task (each agent script's--max_steps); left empty it uses the chosen agent's own value, declared with the newbykey.Touches
app/nullius/standard/:run.py,relay.py,container.py,job.py,cli.py,recipe.py,definitions.pyapp/nullius/runs/classify.py:UnitCallCapErroris an agent errorstandard.json, their drivers andupstream-settings.jsonapp/static/builder/: the Limits stepdocs/cli/run.md,docs/standard.md,docs/builder.md, the field guide and the generated supported-benchmark pagesBefore and after
The Limits step for τ³-bench, before and after:
Verified
main, with feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124's serving providers and feat(standard): run each unit's agent in a Boat VM of its own #150's Boat VMs).max_calls=3.--max-calls.--max_stepsis replaced, or added when the agent's command doesn't set it.Risk
--timeout agent=SECONDS, or by revertinglimits.turn_seconds.run.pyand the catalogue, so expect merge conflicts.