Repository navigation
feat(standard): let a benchmark keep its tasks gated, pin its own providers, set its own limits and prove a merged null control - #133
Conversation
Closes #136 ## What On Runs → By launch, an agent or control with 2 or more shards is now one row, when `cli merge` would merge those shards. The row shows the numbers of the merged job. `launches.fold` makes that job in memory: the trials of every shard, and the job record from `merge.result_of`. `result_of` is the code that `merge` used inline to compute the job record, now a shared function. `merge.check` keeps the trial records that it already reads. - The rate is over every trial with a verdict, never an average of shard rates. - Units that no shard ran come from `merge.check` and stand beside the count ("2 of 8 not run"). They are never misses. The notes of `merge` are the tooltip. - The state is the most telling state of the shards, with a warning such as "1 of 2 shards stopped". - The header gives the dataset size, not the size of one shard. The row is folded by default. A click on the row, or Enter or Space on its chevron, unfolds one row per shard ("shard 2 · host-2"), each with its Open. The fold uses the shared chevron and slide of `disclosures.js`, and respects reduced motion. A refresh of the list keeps an unfolded row unfolded. Controls fold the same way, stay unranked and stay out of Compare. The task-check line is judged over every shard: "Task checks incomplete: oracle not run on shard 2." A participant with one shard keeps its plain row. Shards that `merge` refuses also keep their rows. Job pages, `cli merge` and merged jobs do not change. The header counts every shard ("8 runs"), every shard has its row, and each folded row says how many shards it holds ("2 shards"). A fold reads every trial record of its shards. It is kept in memory for the shard jobs that the workspace cache gives, and it is read again only when the workspace parses a shard again. A new set of shards replaces each kept fold that shares a shard with it, so the cache holds one fold per shard at most. ## Touches backend · the standard (`merge.py`) · routes · templates and static · docs and site ## Before and after One launch over 2 machines, 8 units: 2 agents, nop and oracle. Shard 2 stopped while terminus-2 ran (2 of its 4 units), and the oracle did not run on shard 2. Light theme, 1280 px. | Before | After, folded | | --- | --- | | <img width="980" height="670" alt="1-before-shards-apart" src="https://github.com/user-attachments/assets/990ceda8-8212-4003-ab1c-f380be783fc9" /> | <img width="980" height="421" alt="2-after-folded" src="https://github.com/user-attachments/assets/f37d9af1-7b3b-4b05-abc6-7dca2da1c983" /> | Before: 8 rows, 2 rates for each agent, "4 tasks", "oracle not run". After: 4 rows, "8 tasks". terminus-2 shows 50% and "3/6 · 2 of 8 not run" with "1 of 2 shards stopped". The task-check line reads "Task checks incomplete: oracle not run on shard 2." | After, unfolded (click and keyboard) | After, dark, 390 px | | --- | --- | | <img width="980" height="595" alt="3-after-unfolded" src="https://github.com/user-attachments/assets/b30f3189-d737-4658-8701-a3bbc434ad4a" /> | <img width="358" height="844" alt="4-after-dark-390" src="https://github.com/user-attachments/assets/6c4b106c-218b-4308-a934-ecac1f96934d" /> | ## Verified - Backend test `Fold.test_a_fold_reads_as_the_merged_job`: three shards, one absent, and one with a planned unit that did not run. The folded job equals `parse_job` of the `merge.merge` output on every field the row shows (solved, rates, outcomes, planned, errors, start, end, time, tokens, cost, mean score, budget). The rate is not the mean of the shard rates. The units not run and the notes are those of `merge`. Shards that `merge` refuses are not folded. - Backend test `Fold.test_a_fold_is_read_again_only_when_the_workspace_parses_a_shard_again`: a second fold of the same cached jobs does not call `merge.check`. After a trial record changes, it calls `merge.check` again. After a third shard appears, the cache holds only the fold of the three. - Route tests `test_launch_shards`: 8 shards give "8 runs", 8 shard rows and 4 folded rows that each say "2 shards". The rows are folded by default. Each shard row has its Open, and the folded row has none. The rate and time of the folded row equal those of the merged job. "2 of 8 not run" and "1 of 2 shards stopped" show. The controls fold and stay out of Compare. A nop that passes one task on shard 2 gives "A control did not behave". A stopped nop shard gives "nop stopped on shard 2". Participants with one shard keep their plain rows. - On `c7af06d5`: backend suite `python3 app/tests/parallel.py` (1989 tests) and route suite `python3 app/tests/parallel.py app/ui_tests` (328 tests) pass in a clean container. - Browser, with the fixture above: folded and unfolded, light and dark, 1280 and 390 px. No page errors and no horizontal page scroll. A click on the row unfolds it, and the shard rows slide open and closed. Tab moves the focus to the chevron of the oracle, Enter unfolds it, and Space folds it again. With reduced motion, the rows show at once. At 390 px, a long agent name wraps and the chevron stays on the line of the name. The name, its "2 shards" line and the shard rows start at the same left edge. - Render time of `/runs?view=launch`, median of 15 requests after a first one, with the same parsed jobs given to the page. 2 agents and nop and oracle, each 4 shards of 500 units (8000 trials): 71 ms before this change, 101 ms after. The first render reads the trial records once: 540 ms, against 199 ms before. The rest of the added time is the outcome bars of the 4 folded rows, over 2000 trials each. 8 shards of 1 unit each (32 trials): 11 ms before, 13 ms after. ## Risk If the fold is wrong, a row shows numbers that differ from the merged job. The backend test compares the two directly. `merge` writes the same `result.json` as before through `result_of`, and its tests pass unchanged. The first render of a large sharded launch reads all its trial records once (+0.3 s for 8000 trials). Later renders use the cache until a shard changes. The cache holds one fold per shard at most, with the parsed jobs that the workspace cache also holds. Open pull requests #124, #133 and #134 also change `merge.py`: the second to merge must rebase. Revert the commit to undo.
…l everywhere (#148) Closes #147 ## What Each model provider is now one record that every feature reads. A feature asks for the protocol that it speaks, and gets the providers that serve it. - **The registry** (`app/nullius/providers.py`). `PROTOCOLS` defines each wire format once: its default path and how the key is sent (`chat`, `responses`, `messages`, `systemone`, `decisions`, `openai-decisions`). `PRESETS` holds OpenRouter, OpenAI, Anthropic and TypeSafe. `record()` returns `{name, base, key, endpoints}` for a preset or a custom provider. `key` is the name of the secret setting, never the key. - **Storage** (`app/nullius/settings.py`). A record is one JSON setting of a new kind, `provider`. A custom provider is `MODEL_<ID>`. The endpoints added to a preset are `OPENROUTER`, `OPENAI`, `ANTHROPIC` or `TYPESAFE`. For these settings the environment does not replace the file. Its value is a partial object, merged over the file's value (RFC 7396, `null` removes a field). Settings shows read-only only the fields that the environment sets. Keys keep their names and stay separate secrets. - **Migration.** `MODEL_<ID>_NAME` and `MODEL_<ID>_API_URL` are still read as `name` and `base`. Saving a card moves them into the record. `nullius migrate --providers` previews that rewrite of `settings.json`, and `--apply` writes it. Recipes do not change. `local` and `local:<ID>` still read as `custom`. - **Consumers.** The registry is read by the builder, the provider test, onboarding, `execution`, standard runs, the relay, the CLI's `--provider` and both judges. The builder's JavaScript copy of the list is gone. The builder offers a provider only where it serves what the recipe's agents and judge speak, so Anthropic appears for an agent of the standard that speaks `messages`. Review, Run (`web/execute.py`) and `run.Run` refuse a provider that does not fit. The relay sends `x-api-key` and `anthropic-version` on a `messages` route only, and a bearer header on a benchmark's own routes. - **Settings cards.** A card shows Base and a collapsed Endpoints section with one row for each protocol. The placeholder is the default URL. A base on a preset's host suggests that preset's endpoints. A preset's known endpoints are fixed. An endpoint added to a preset is a path, or a full URL on the preset's own host, because the preset's key goes there. - **System One** (`app/nullius/judge/client.py`) asks each decision protocol: the TypeSafe shape on `systemone` and `decisions`, and its own adapter for `openai-decisions`. It keeps the provider's `usage.cost` and records a `refusal` as no answer. A 402 says what it means. A judge never follows a redirect with the key. - **Judge cards.** Each judge has Provider, then Protocol when the provider serves two protocols of that judge (else one quiet line, "via ..."), then Model. Each judge's provider is a select whatever the count, because the list is the user's own providers and grows or shrinks with them; cards are kept for fixed sets of choices. New settings: `LLM_JUDGE_PROTOCOL`, `SYSTEM_ONE_PROTOCOL`. On OpenRouter, System One offers OpenRouter's public list of decision models, asked without a key, cached 6 hours, with the fixed list as the fallback. The seven commits are atomic and in reading order: registry and storage, consumers, Settings cards, decision protocols, judge cards, tests, docs. ## Touches backend (`providers`, `settings`, `migrate`, `onboarding`, `execution`, `judge/client`, `recipes/launch`) · the standard (`run`, `relay`, `recipe`, `cli`, `definitions`, `fidelity`) · routes (`web/settings`, `web/execute`, `web/filters`) · templates and static (Settings cards, judge cards, builder core) · docs (`docs/settings/model-providers.md`, `system-one.md`, `llm-judge.md`, `README.md`, `agent-keys.md`, `docs/builder.md`, `docs/judgement.md`, `docs/standard.md`, `docs/cli/README.md`, `AGENTS.md`, `app/.env.example`) ## Before and after Settings at 1280 px, light theme. Fixture keys only. Model providers. TypeSafe is a preset like the others, so it comes after the run providers. | Before | After | | --- | --- | | <img width="748" height="431" alt="1-before-providers" src="https://github.com/user-attachments/assets/1dd2338e-0b32-4631-8895-a6df2ba7fc29" /> | <img width="748" height="431" alt="2-after-providers" src="https://github.com/user-attachments/assets/ff3bb210-22ef-4bff-bbe4-df4ac4488519" /> | A preset card (OpenRouter), Endpoints open. Known endpoints are fixed. Other protocols are editable rows. | Before | After | | --- | --- | | <img width="748" height="360" alt="3-before-preset-card" src="https://github.com/user-attachments/assets/5fb776e9-1c86-4cf0-8508-24041b6f9042" /> | <img width="748" height="706" alt="4-after-preset-card" src="https://github.com/user-attachments/assets/9058e8e4-339e-49da-8a44-67b2ffcdf8bf" /> | A custom provider, Endpoints open, with one endpoint set. | Before | After | | --- | --- | | <img width="748" height="451" alt="5-before-custom-card" src="https://github.com/user-attachments/assets/c725f642-3d22-4b8d-8fb0-c702e415594d" /> | <img width="748" height="897" alt="6-after-custom-card" src="https://github.com/user-attachments/assets/afcbcb01-fed9-4ef3-87c2-883d73594edc" /> | Judges. System One on OpenRouter serves two of its protocols, so it shows Protocol. The LLM judge on OpenAI serves one, so it shows "via OpenAI Chat Completions". | Before | After | | --- | --- | | <img width="748" height="504" alt="7-before-judges" src="https://github.com/user-attachments/assets/a089a1bd-9f72-4ca6-bff1-e1a652a83ec1" /> | <img width="748" height="503" alt="8-after-judges" src="https://github.com/user-attachments/assets/61ef9ae3-e080-4e54-99a1-dc43d4960ba6" /> | The custom card at 390 px: <img width="358" height="1153" alt="9-after-custom-card-390" src="https://github.com/user-attachments/assets/dce35d02-44b1-4ccb-8362-c84129e1eb87" /> ## Verified The branch is based on `main` at 1cc37ac. I tested its tip, each suite in a new rootless Podman container (`python:3.12`, `--init`, uv on PATH, Node for the route suite): - Backend suite (`python3 app/tests/parallel.py`): 2029 tests in 118 modules, OK. `main` at 1cc37ac, in the same container: 1992 tests in 116 modules, OK. - Route suite (`python3 app/tests/parallel.py app/ui_tests`): 441 tests in 42 modules, OK. `main` at 1cc37ac: 431 tests in 41 modules, OK. - New modules: `test_provider_registry` (17 tests: the merge, `null`, partial environment values, the old settings, record checks, the preset host rule, keys never in a record), `test_judge_decisions` (18 tests: each decision protocol's request and reply, `usage.cost`, `refusal`, 402, no redirect, the cached model list and its fallback), `ui_tests.test_judge_cards` (2 tests: the protocol row and the live list). Existing modules gained tests for the relay's headers, the builder's provider filter, Harbor recipes on Anthropic and the Settings cards. - End to end, against the server (`app/app.py`) and the CLI, in a container with no network, with a new `NULLIUS_HOME` for each case and fixture keys only. 37 checks, all pass: - No settings: the four presets and the OpenAI-compatible card. - A custom provider in `settings.json`: its card, its merged record, the default path for a protocol without an entry, System One offers it for its `decisions` endpoint, no key in the page. - `MODEL_GW='{"base": "https://gw-eu.example/v1", "endpoints": {"decisions": null}}'` in the environment: the base changes, the name stays the file's, the endpoint goes, System One no longer offers it. The card shows Base read-only and Name editable. The file does not change. - `nullius migrate --providers`: the preview names the moves, prints no value and leaves the file as it was. `--apply` writes the record, removes the old lines, keeps the key and the file mode 0600. A second run finds nothing to move. - Old variables in the environment (`MODEL_API_URL`, `MODEL_<ID>_NAME`, `MODEL_<ID>_API_URL`, `LLM_JUDGE_PROVIDER=local:<ID>`): read as before. - System One's model list on OpenRouter, offline: the route answers with the built-in list in less than the 5 s timeout, or with a cached list, however old. - Browser (Chromium, Playwright), on this branch and on `main`, at 1280 px and 390 px, light and dark, with no network: the provider list, a preset card and a custom card with Endpoints open, and both judge cards. No page errors and no horizontal scroll. Not verified: live calls to OpenRouter Decisions and OpenAI Decisions, and a whole run on Anthropic. The tests use fixtures. ## Risk A wrong record or header can send a call to the wrong URL, or with the wrong key header. Undo with a revert. The old settings are still read, so a revert keeps working with them. Records written by a card stay in `settings.json` and the older code ignores them. - **Migration on edit.** Saving a custom provider's card moves `MODEL_<ID>_NAME` and `MODEL_<ID>_API_URL` into `MODEL_<ID>`. After a revert, that provider loses its name and base until they are set again. `nullius migrate --providers` without `--apply` shows what would move. - **New reserved names.** `OPENROUTER`, `OPENAI`, `ANTHROPIC`, `TYPESAFE` and `MODEL_<ID>` are now settings. An agent key listed under one of these names is left out, and Settings says so. A shell variable such as `MODEL_PATH` that is not a JSON object is not read as a record. - **`--provider anthropic` on the CLI.** The key header now follows the agent's protocol. An agent that speaks only `chat` is refused on Anthropic, where it used to get `x-api-key` on OpenAI-style paths. - **Runs and custom endpoint paths.** The relay forwards the agent's own path below the provider's base. An Endpoints row changes where the judges and the test ask, not where a run's calls go. - **`openai-decisions` field names** come from OpenAI's documentation, not from a live reply. - **Open PRs on the same files.** This change does not depend on any of them. Whichever merges second rebases. A test merge of this branch with each one: - #124 (OpenRouter serving providers, `--served-by`): conflicts in `providers.py`, `standard/run.py`, `standard/relay.py` and `docs/settings/model-providers.md`. Each conflict is small. Both add functions at the same place in `providers.py`. This branch removes `run.PROVIDERS`, where #124 adds `routing` to the OpenRouter entry: that flag becomes a field of the OpenRouter preset. Both change `Relay._headers`. - #133 (stacked on #124): the same four files, and also `standard/fidelity.py` (one `key_header=` line, now `r.key_header`) and `docs/standard.md` (one table row each). - #134 (stacked on #133): the same as #133, and also `ui_tests/test_settings_page.py` (one placeholder assertion, which both change). - #124, #133 and #134 also conflict with `main` alone in `standard/merge.py` (and #133, #134 in `tests/test_standard_shard.py`), which this branch does not touch.
d19a802 to
a6d32e7
Compare
| raise Refused(f"--served-by names {', '.join(stray)}, which this run does not call: it calls " | ||
| + (", ".join(called) or "no model")) | ||
| self.route_pins = pins_of(bench, other, provider, self.judge_model) | ||
| self.serving.update(self.route_pins) |
There was a problem hiding this comment.
The definition's pins override Served by here, but the builder never shows them.
self.serving.update(self.route_pins) makes setups.original.pin and score.judge.pin win over the user's --served-by for the same model. That is the intent. However, the builder's Served by (#124: command-builder.js, recipe.py served_by) does not know these pins exist. It still says "any provider" and offers every provider for a pinned model, so the page offers a choice the run then ignores.
Reproduced with OSWorld2 (#134, which pins MiniMax-M3 to minimax and the official judge to anthropic): choose Like the paper, OpenRouter, MiniMax-M3. Served by lists 8 providers (CoreWeave, DeepInfra, GMICloud, Minimax, Parasail, ...).
| Served by in the builder | What the run sends |
|---|---|
| empty ("any provider") | minimax only |
| DeepInfra | minimax only, with no notice |
| the judge's Served by, anything (official judge) | anthropic only |
Review does not say the user's pin will be replaced.
Suggested fix:
- the recipe compile (which already returns
served_models) also returns each role's definition pin, computed by the same rule aspins_of; - the builder draws a pinned role's Served by as a fixed value (the paper's providers), with no menu;
- the recipe check, and
--served-byon the CLI, refuse a pin that disagrees with the definition's for that model, instead of overriding it silently.
This could land here or in #134. #134 is where it first becomes visible, since OSWorld2 is the first entry with a pin.
There was a problem hiding this comment.
This also clashes with the design of #124, which added --served-by. There, the serving provider is the user's choice: nothing is pinned unless the user asks, and an empty Served by means any provider. This PR has the definition pin some models whatever the user picks. Those are two different rules for who decides the provider.
Before choosing a fix, the authors of #124 (pierreleca) and this PR (sirotaoleg) should agree on one rule. For example:
- the definition's pin is only a default, shown in Served by, that the user can change; or
- the definition's pin is fixed for the Original and the official judge, and Served by shows it as fixed for those models.
… work on a unit and cap its model calls (#191) Closes #190 ## What A standard run can now force its limits, and the agent's time limit covers its whole work on a unit. - **Time limits by phase.** `run --timeout PHASE=SECONDS` forces the limit of a phase: `setup`, `image`, `agent`, `verify` or `score`. Empty keeps the default, which comes from the task, then the benchmark, then Nullius. `config.json` records what was forced under `standard.timeouts`. - **One agent limit per unit.** The agent's limit is one deadline for its whole work on a unit: every turn of a conversation, and the simulated user's time between them. Before, it applied to each reply. A capability keeps one limit per add and per recall, because a memory adds a unit's sessions one by one. τ³-bench's `limits.turn_seconds` goes from 600 s a reply to 7200 s a conversation. - **Turns.** `run --max-turns N` sets the turns of a conversation. The builder offers it only on a benchmark that holds conversations. - **A cap on model calls.** `run --max-calls N` caps the model calls an agent makes on a unit. The relay counts each answered call on the agent's route; rate-limited and failed calls are not counted, and neither are the judge's or the simulator's. Past the cap the relay refuses the call with a 402, and the unit ends as `UnitCallCapError`, an agent error that is not scored. It is for agents only: a capability's route also carries the benchmark's reader, so a capability run refuses it. - **A benchmark's own bounds.** They are now options marked `limit`. τ³-bench gets Steps per conversation (`--max-steps`, from 1 up to its default of 200) and Tool errors per conversation (`--max-errors`). OSWorld gets Steps per task (each agent script's `--max_steps`); left empty it uses the chosen agent's own value, declared with the new `by` key. - **Builder, Limits step.** It shows Scope, then Time limits in minutes, then a section named after the benchmark when it has bounds of its own. An empty field shows the value the run uses: in the text colour when the benchmark or the agent declares it, grey when it is Nullius's default (priority: Nullius < agent < benchmark). Nothing is typed into a field for the user. ## Touches - `app/nullius/standard/`: `run.py`, `relay.py`, `container.py`, `job.py`, `cli.py`, `recipe.py`, `definitions.py` - `app/nullius/runs/classify.py`: `UnitCallCapError` is an agent error - Catalogue: τ³-bench and OSWorld's `standard.json`, their drivers and `upstream-settings.json` - `app/static/builder/`: the Limits step - Docs: `docs/cli/run.md`, `docs/standard.md`, `docs/builder.md`, the field guide and the generated supported-benchmark pages ## Before and after The Limits step for τ³-bench, before and after: | Before | After | | --- | --- | | <img width="440" alt="Limits step on main: Scope only, with tasks, concurrency, attempts and two spending caps" src="https://github.com/user-attachments/assets/80b468a4-326f-4057-a4b2-5551ea93e642" /> | <img width="440" alt="Limits step on this branch: Scope with turns and model calls, Time limits in minutes, then a tau3-bench section with steps and tool errors" src="https://github.com/user-attachments/assets/6278494c-6c9f-456f-b739-9578494315ee" /> | ## Verified - Backend suite: 2175 tests OK. Route suite: 531 tests OK (after merging `main`, with #124's serving providers and #150's Boat VMs). - New tests: - A whole conversation that ends past the agent limit, though no single turn does. - Four one-call turns cut on the fourth call by `max_calls=3`. - The relay counts only answered calls on the agent's route. - A slow capability whose three adds each fit a 1 s limit while their total does not. - A capability run refuses `--max-calls`. - Recipe validation of the limits, including a value that rounds to 0 s. - τ³-bench steps above 200 are refused. - OSWorld's `--max_steps` is replaced, or added when the agent's command doesn't set it. - Against the installed τ³-bench upstream: the driver uses 200 steps and 10 errors by default, and 7 and 3 when the run sets them. - Checked in the browser, light and dark, for τ³-bench and OSWorld. ## Risk - **Agent timeouts on conversation benchmarks.** A conversation that used to fit 600 s per reply now has a single limit for the whole conversation, 7200 s on τ³-bench. A very long conversation can now time out where it didn't before. Undo by forcing a larger `--timeout agent=SECONDS`, or by reverting `limits.turn_seconds`. - **Comparability.** A run whose limits differ from the defaults is not yet marked as not comparable with the leaderboard. - **Overlap.** #174 (the agent's setup timed apart from its run) and #133 (a benchmark's own limits) change the same parts of `run.py` and the catalogue, so expect merge conflicts.
d234aa8 to
62bec8f
Compare
|
The definition pins now show in the builder, but under Any of your choice too: OSWorld2 opens with |
…on top of --served-by: gated tasks, a definition's pins as its serving pins, a pinned route's refusal as the unit's, limits of their own, and prove of a merged control A definition's pin (setups.original.pin, score.judge.pin) joins the run's --served-by pins and wins over --served-by for its model; ApiProviderRouteError reads the pin and reply.served_by. A failed Responses object reads reply.end and reply.error.
…st 429 A unit whose driver scored it after a guard's stop keeps that score, marked stopped, though its last own call got a 429.
… own last 429, not a side call's or a context window's stop The rule reads the agent's own last call, a guard's line in the stop file and a score stopped by other than the context window; a cap's refusal is still the cap's.
…h --served-by replaces, as ThinkFlowLab#124 has the user choose A model the definition pins (setups.original.pin, score.judge.pin) goes to the user's providers when --served-by gives it others: the run says so, records the definition's under standard.pins_replaced, and the Original's run does not compare with the paper. The recipe's compile gives each field's default (served_pins) and sends a field's list, its own or the default, with every job that calls its model; the builder's Served by starts with it, and Review says when a field's differ. --served-by without MODEL= names the model an Original sends, and prove --job refuses a control whose pins were replaced.
…tcome and prove text and their messages
…r place, the recipe keeping the value; OSWorld1's agents by their upstream's names
…e the work runs on Where, a single value as fixed, and what the datasets need on Dataset; OSWorld1's VM provider on Where
…eclares, with their route errors; Served by is the user's alone, as ThinkFlowLab#124 has it
62bec8f to
e768c63
Compare
…ces on the machine and a null control (#134) Stacked on #133: this pull request's own commits are the last sixteen. Until #133 is merged, the diff shows its files too. It can be merged after #133. ## What OSWorld-V2's 108 tasks become a bundled benchmark, `osworld2`, on the Nullius standard, at the upstream's release `osworld-v2.1`. Each task is a unit in an Ubuntu VM of its own on the upstream's docker provider, set up by the task's own class and scored from 0 to 1 by its own evaluate(). The tasks and their assets are gated on Hugging Face. Nullius does not fetch them, and the entry is `gated` (#133): a unit's failure from a model endpoint says its status alone, and the driver names an upstream exception by a known class's name at most, never by its words. The Original setup runs one of OSWorld-V2's own agents, M3 or Muse Spark, chosen by the Agent option, from its published script, with every model call sent to the relay. Claude Sonnet 4.6 is the LLM grader of 21 tasks and the simulated user of 6. No provider is pinned: Served by is the user's, as for OSWorld1. Only M3's runs count as comparable with the paper's row. Its Review row and its generated page name the setup "OSWorld-V2 agents". A null control runs one `DONE` action, then the task's evaluate() on the untouched VM. The entry declares `agent.computer/v1`. No agent from the Agents page runs it yet. Its texts are one Good to know line and 9 one-line limitations. The mock websites of 40 tasks are OSWorld-web at the same tag. The guide's steps build them from source and serve them on the machine the units run on. A unit that needs them is refused before its VM starts when they do not answer. For 026 and 041, Nullius makes GitLab on the run's machine before the first of them, with a root token it keeps there, then starts it anew from its snapshot for each, one task at a time; Overleaf's 2 tasks get Overleaf started anew. Each of the 8 tasks that ask for a proxy gets a proxy of its own. The options are the agent, by name, Thinking for M3, and disk write limits for each VM. The website host suffix is asked on the Dataset step, for the four datasets with website or GitLab tasks. Steps per task is with the limits, each agent's own by default, as its wrapper runs it: 100 for M3, 500 for Muse Spark. The VM provider, Docker alone, is on the Where step. By default M3 takes 100 steps with thinking off, as the upstream's M3 script does; the paper ran M3 at 500 steps with thinking on. Muse Spark runs as published at this release, while its published score came from newer upstream code; a limitation says it does not compare, and a docs page says what differs. There are six datasets: All tasks (108, the upstream's `test_v2.json`), Offline (82), and four by the services their tasks need. Tasks 103 and 104 are not supported: their evaluate() needs FreeCAD on the driver's machine. Review and the run's start ask for Docker and for the scoring programs (ImageMagick, ffmpeg, tesseract) on the driver's machine. As for OSWorld1, the Where step greys out the container card under the Original, whose docker provider needs Docker on the machine. A guide, `docs/benchmarks/osworld2-machine.md`, prepares a machine, runs the null control on one task as its last setup step, and says what a run takes in time and money. A second page, `docs/benchmarks/osworld2-results.md`, says how a run differs from the paper and the upstream, and how a unit is scored, stopped or left unscored. ## Touches standard (an option's example and a machine setup's label: the definition check, the benchmark and agent pages, the builder, Review) · catalogue · templates and static (the mark) · docs and site (`docs/benchmarks/`, the generated pages, `AGENTS.md`) · routes (tests only) · packaging (the entry's `.yml`, `.Dockerfile` and `startup/*.py` as package data in `pyproject.toml`) ## Before and after **Before:** The Benchmarks page before this change  **After:** OSWorld2 listed, with its mark  **Before:** The builder's Benchmark step under Computer use: OSWorld1, beside the selected GAIA  **After:** OSWorld1 and OSWorld2  **Before:** The Agents page: "n/7 benchmarks it runs"  **After:** "n/8": OSWorld2 counts, and no agent runs it  **Before:** An agent's page, Codex: the benchmarks it runs  **After:** OSWorld2 listed: it needs `agent.computer/v1`, which Codex does not offer  **New:** OSWorld2's page: its setup and its proofs by dataset  **New:** Its options: the agent by name, steps, thinking, the VM provider, the website host suffix and the disk write limits  **New:** Its dataset and its Good to know line  **New:** Its limitations  **New:** Dataset step: six datasets with their controls; All tasks chosen asks for the website host suffix, and for no token  **New:** Agent step: OSWorld-V2 agents; the Agents page's agents lack computer use; the settings every agent shares, the disk write limits  **New:** The Original's own settings: MiniMax M3 or Muse Spark by name, and thinking  **New:** Muse Spark chosen: Thinking, an M3 setting, is gone  **New:** Model step, Like the paper: MiniMax-M3 on OpenRouter, Claude Sonnet 4.6 as the judge, each served by any provider until one is named  **New:** Limits step: Steps per task, each agent's own by default, 100 for MiniMax M3  **New:** Where step: the VM provider, Docker alone; no container or Boat backend here; Docker and the scoring programs checked on this machine  **New:** Review: every decision, both prerequisites ready, no model key in this fresh profile  **New:** The null control's merged job on test_v2, its first rows  **New:** 052 by the test stand's model in M3: 1.0  Taken at 1400 px, light theme, on a fresh profile. Cut to part of their page: the Benchmark step and Agents pairs (the top 1000 px), Codex's page (the top 1100 px), OSWorld2's page (four parts) and the null control's job (its first rows). The two jobs are runs named under Verified, opened in the app. The Model step, Review, OSWorld2's options, dataset and limitations, and the Dataset, Agent and Where steps were taken again after main's Served by (#124), Boat (#150) and time limits (#191), the shorter texts and GitLab made by Nullius, and checked at `273cca5e`: the other pictures differ there only in the sidebar, which they leave out. The two jobs were taken again at `c4427915`, on main at `0d83e393`, whose job page names the judge's button anew (#196); the other pages and steps pictured are the same there. OSWorld2's options and limitations and the Dataset, Agent, Model, Limits, Where and Review steps were taken again at `9526a5b1`, after the agents' names, the steps as a limit, the VM provider on Where, the website host suffix on Dataset and Served by with no default. ## Verified - At `9526a5b1`, on #133 (`e768c632`) on main at `db6e8f96`, on Python 3.12: backend suite (`python3 app/tests/parallel.py`), 2688 tests in 144 modules, OK; route suite with Node on PATH, 643 tests in 53 modules, OK; the wheel's check (`app/packaging/verify.py`), OK. - **OSWorld2's 15 test modules,** 375 tests, OK; they need the entry, so none passes without it. 10 are skipped here: they run only when the environment names the upstream with its gated task classes, OSWorld-web or GitLab. Three route test modules and three backend test modules changed for the new entry: the benchmark lists and counts, the Where step's container card, Settings' custom keys, and OSWorld2's case of gated tasks (`test_standard_gated.py`). Option examples and setup labels change `test_standard_options.py`, `test_prerequisites.py` and `test_standard_bench_options.py`, and the route modules `test_catalogue_routes.py`, `test_agents.py` and `test_builder_where.py`. - At `9526a5b1`, in a headless Chromium, light and dark, 1400 and 390 px: the builder's Agent step (the agents by name, Steps no longer there), Model step (Any of your choice: no provider filled in), Limits step (Steps per task, 100 for M3 and 500 for Muse Spark), Where step (VM provider docker, fixed) and Review; no page error. - In a headless Chromium, light and dark, 1400 and 390 px: the Benchmarks page, OSWorld2's page, the Agents page, Settings and the builder, no page error or cut text. OSWorld2's limitations fit one screen at 390 px, light and dark. `/ui-sweep --only osworld2` at `fc0bdb1f`, this pull request's third commit before the rebases, dropped since (it set `limits.run_again`, which main's `--retries` replaced): 6 screenshots, none with a page error or cut text. - `PYTHONPATH=app python3 -m nullius standard check`: ok. `python3 app/tests/catalogue_docs.py --write`, then the check: up to date. `python3 app/tests/field_contract.py`: ok. `app/site/readme_media.py --check`: current. The docs check reports the same two findings in `AGENTS.md` as on main. - **Where the runs below ran.** The commits named here are of the port's working branches, not in this repository: - `c3637ffd` is this pull request's tree with main's two later commits merged in; - `f74d1a00` and `58c2b586` add one test commit: two JSON files, no code. It gives OSWorld2's Original one more model, MiniMax M3.1 Flash, the name it is served as, its pin to MiniMax and `--max_tokens 32768`. That model is a test stand, not in this pull request; - `58c2b586` is this entry on the core before #124; - `34e35370` and `8e2e381b` are of the port before its move to this repository's history. - **The controls,** recorded in `standard.json`'s `proven`: - Offline, no services, on 2026-11-04, on an earlier commit: the 36 ids are the same twice, and the null control scores 0 on 4 tasks; - All tasks, on 2026-11-15, on `fc0bdb1f`, this pull request's third commit before the rebases, dropped since (it set `limits.run_again`, which main's `--retries` replaced); the next commit only writes this record, and the one after changes three tests. The null control ran on all 108 tasks, one launch with each task a shard of its own, at 16 MB/s and 160 IOPS a VM, in six lanes with at most seven VMs at a time; it took 1 h 32 min. 108 and 093 failed their setup under that load and passed alone at 24 MB/s and 240 IOPS, in 24 more minutes. The grader cost $0.14. `nullius standard prove osworld2 --dataset test_v2 --job nop=<merged job>` passed: the 108 ids are the same twice, 0 of 108 at full score, a mean of 0.0021 against a bound of at most 0.01. Five tasks score above 0 on the untouched VM, as a limitation says; - every task is in line with the same control on all 108 tasks on `34e35370`, which had the same mean; 077's LLM grader gave 0.054 both times. - **GitLab made by Nullius,** on a server whose GitLab had been set up by hand: the null control of 026, alone, made GitLab first (cloned, configured, its root token, its runner, its snapshot) and scored 0.0 in 15 min; 041, alone, started from that snapshot and scored 0.0 in 7 min; a snapshot of an older stamp was made anew by itself. No model call, no GitLab left after either, and the token in no file of the jobs. - **An agent's score on every kind of task,** each above the same task's null control (0.0 in each): - **Seven kinds,** on `58c2b586`, by the test stand's model in OSWorld-V2's M3 agent, from the command line: - desktop apps, 045: 1.0; - mock websites, 052: 1.0; - Overleaf, 072: 0.775; - GitLab, 026: 0.5, a partial score; - a proxy, 075: 1.0 (recorded 0.9999999999999999), with the unit's proxy carrying new outbound connections while the agent acted; - the public internet, 027: 1.0; - the LLM grader, 003: 1.0, with both grader calls answered 200. Each of the seven gives its recorded score again at `c3637ffd` when its outcome is computed from its saved records, with no VM and no evaluate(). - **The eighth, the simulated user:** 007, 0.8, by Muse Spark on `8e2e381b`; the simulated user answered 7 times. It was not run on this base. - **Live on this base, against OpenRouter:** - on `c3637ffd`, the null control on 087, a graded task: 0.0. The grader's call went to Anthropic by the definition's pin, and `standard.served` reads `held: true`; - on `f74d1a00`, the test stand's model on 052: 1.0. All 68 calls went to MiniMax by the test commit's pin and named it, `held: true`, with no error. - **This pull request's own agents** have no positive score on this base. M3 ran one task here, 017, on the core before #124 (`90fde73e`), and scored 0.0: the agent declared the task failed at its 120th step. Muse Spark ran none. Their scores before the move, on `34e35370` and earlier: - M3 on 9 tasks: 070 1.0 twice, 030 1.0, 045 1.0, 013 1.0 then 0.0, 031 0.714, 017 0.429, 010 0.0 three times and once an environment error; 026 and 098 left no score (a full context window, a spend guard); - Muse Spark on 3 tasks: 045 1.0, 007 0.8; 052 left no score (a launch error of ours, then a spend guard). - **Every attempt before the move,** two test stands' models included, was recorded and triaged: 61 attempts on 33 tasks. 43 were scored (21 full, 11 partial, 11 zero), 10 ended in an environment error and 8 were stopped by a guard or by hand. None was an agent error. Each of their full and partial scores was checked for the false-success patterns the port's audits found: none. - Not run: a positive score of M3 or Muse Spark on this base; the simulated user on this base; task 041 with an agent; a parity run of the paper's 108 tasks; the 10 skipped tests with the upstream named; the docker tier; the site build. ## Risk - OSWorld2's driver and OSWorld-V2's scripts run only when someone runs OSWorld2. The Agents page counts eight agent benchmarks, and the generated agent pages gain an OSWorld2 column. No agent there declares `agent.computer/v1`. - **Scores.** A run compares with the paper's row by its model alone: Docker instead of AWS, no US residential proxy, release `osworld-v2.1`, the grader through OpenRouter, and by default 100 steps with thinking off. A wrong error kind would move a unit between "environment error" (left out of the adjusted rate) and the agent's error (counted). - **Steps.** A Muse Spark run that sets no step limit now takes up to 500 steps, its wrapper's budget, where it took 100: a longer and dearer episode. M3 keeps 100. A saved recipe or command that names a number keeps it. - **The machine.** - VMs, the mock websites, GitLab and the proxies run on the machine the units run on, outside a run's constraints in Settings; `-j` bounds them. - GitLab's Docker in Docker and the websites' image build run as privileged containers. - In each of the driver's processes, `startup/sitecustomize.py` wraps the creation of the VM's container: its network, its ports on 127.0.0.1, its disk and its write limits. The VMs need a free Docker subnet. - The VM's clock is UTC+8: a dated task set up before 16:00 UTC and acted on after can score 0. - The first unit of 026 or 041 on a machine clones Task-Web/gitlab at its pin, pulls its images (about 7 GB) and makes its snapshot: about 8 minutes more, once. - **Secrets.** GitLab's root token is made on the run's machine and kept there in a file for its user alone, beside GitLab's clone; only the processes of 026 and 041 get it. Model keys stay with the relay. Tasks 001, 006 and 010 start their mail client offline, unlike the upstream. - Stacked on #133. Once it is merged, this branch is rebased onto main. - Undone by reverting the commits; the catalogue entry goes with them. --------- Co-authored-by: Oleg <247799477+sirotaoleg@users.noreply.github.com> Co-authored-by: Etienne Filhol <etienne.filhol@hotmail.fr>
What
A benchmark of the standard can now say more about how its units run and end, which a computer-use benchmark like OSWorld2 (#134) needs.
gated, the new limits andprove's new options are opt-in. The relay's rules and the rules of how a unit ends change for every benchmark of the standard; Risk lists them.gated: true): a unit's failure that the relay's records explain, a model endpoint's (the agent's, the judge's or the simulated user's), says in itsresult.jsonthe status and class alone, never the provider's words, which can quote the request. The trial's call records keep the provider's words, and an error a driver raises itself is kept as the driver words it.names: {value: name}): the builder's cards and select, Review and the benchmark's page show the name,aboutstays its tooltip, and a recipe and the CLI keep the value. OSWorld1's agents are named as their upstream spells them.step):where, where the work runs, on the Where step, one value as fixed, and in Review's Where row;dataset, what some datasets need, on the Dataset step for those datasets. OSWorld1's VM provider is on Where.limits.score_seconds: how longscoremay take, the benchmark's level below--timeout score=(feat(standard): force each phase's time limit, bound an agent's whole work on a unit and cap its model calls #191); the Limits step shows it as the benchmark's;limits.call_seconds: how long the relay waits for one model call, alsorelay serve --timeout;EnvironmentSetupError:,VerifierError:,CancelledError:orCauseUnknownError:keeps that kind;ContextWindowExceededError, the agent's;scorecan say the stop's reason (verifier_result.stopped);--retries, a unit a guard asked to stop and a full context window are not tried again;OutputLengthExceededErroronly when its last reply has neither text nor a tool call. When the answerer fails and its last reply was a Responses object answered 200 andfailed, that is an API error.answered: 200). feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124 leaves counting an error inside a 200 to a pull request of its own; this one takes only a 200 with no reply in it;stopfile beside the relay's records: the relay notes in it an agent's call refused for the model's context window, and that note refuses nothing. Once a benchmark's guard writes a line of its own there, the agent's later calls are refused unsent, while side models and the judge's and simulated user's routes go on;.work/relay,result.json,verifier/,agent/) aside to.work/earlier.<ns>.nullius standard prove ... --bench-option NAME=VALUEsets the benchmark's own options for the units check and the controls;--job nop=MERGED(ororacle=) checks a merged launch of a control against its bound instead of running it, and every trial's own record;mergeaccepts shards that differ in benchmark options and keeps each shard's inmerged_from;mergerefuses, unless--force, a trial whose record names another job, benchmark, version, control or agent, and a job itself merged despite differences. It lists the shards absent from a merge only when the launch has at most 1,000.agent/trace.error.txt;thinking_tokenscount as reasoning;comparablecan list the models whose runs compare with the paper, and the builder's message for another model says so;Read first:
failure, then_hollowand the stop inRelay._forward(app/nullius/standard/relay.py). Next, the outcome rules inRun(app/nullius/standard/run.py), thennamesandstepinrecipe.compileandbenchOptionsin the builder, the--jobcheck inprove.pyand the trial check inmerge.py. Thentest_standard_gated.py,test_standard_context_window.pyandtest_standard_controls.py. The rest is the definitions' checks, the CLI options, the sidecar's argument and the pages.Touches
the standard (
app/nullius/standard/:relay,run,prove,definitions,merge,job,cli,container,recipe,fidelity,retries) · the builder (app/static/builder/,app/web/builder.py: the score's default on the Limits step, an option's names, and an option on its own step) · the benchmark page (an option's names) · OSWorld1's definition (its agents' names, its VM provider on Where) · docs and site (docs/standard.md,docs/cli/run.md,docs/cli/read-jobs.md,docs/builder.md,docs/settings/model-providers.md)Verified
e768c632, on main atdb6e8f96, on Python 3.12: backend suite (python3 app/tests/parallel.py), 2310 tests in 129 modules, OK; route suite with Node on PATH, 639 tests in 53 modules, OK; the wheel's check (app/packaging/verify.py), OK.5bcfd82d): 9 fail, 2 pass, and 1 does not end.test_standard_context_window: 19 of 20;test_standard_controls: 17 of 43;test_standard_relay: 8 of 58;test_standard_gated: 4 of 6.test_standard_run: 2 of 50;test_standard_trace: 2 of 20;test_standard_parts,test_standard_served_byandtest_standard_units: 1 each.test_standard_sharddoes not end within 4 minutes: one of its merges names 2^64 shards, and main'smergelists every absent one.test_standard_versionsandtest_dependency_surfacepass on both.limits.call_secondsreaches the relay and the sidecar's--timeout. OSWorld2 is the first entry to set it (test_osworld2_muse.py).test_standard_served_bypasses as main has it;write_shardintest_standard_shardtakes feat(runs): fold a launch's shards into one row per participant #137'srewardbeside this branch'sbench_options.PYTHONPATH=app python3 -m nullius standard check: ok.python3 app/tests/catalogue_docs.py --write, then the check: up to date.python3 app/tests/field_contract.py: ok.app/site/readme_media.py --check: current. The docs check reports the same two findings inAGENTS.mdas on main.--timeout; the docker tier; the site build.Risk
ContextWindowExceededError), where it was an API error outside its rate;rate_limited, where its score stood;failedResponses object is an API error, where it was the agent's;proveruns no control when the units check failed, and fails a control job with a unit that has neither a score nor an error;mergerefuses a trial whose record names another job unless--force.db6e8f96, with feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124, feat(standard): run each unit's agent in a Boat VM of its own #150, feat(standard): force each phase's time limit, bound an agent's whole work on a unit and cap its model calls #191, feat(standard): retry an attempt that fails on the infrastructure or the agent, with --retries beside --attempts #154, fix(standard): time the agent's setup apart from its run #174 and the Recipes page (feat(builder): add a Recipes page for the saved recipes #205). feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124's--served-bycode is main's as merged; feat(standard): force each phase's time limit, bound an agent's whole work on a unit and cap its model calls #191's time limits come first, so--timeout score=is used overlimits.score_seconds, and a unit's cap on model calls (UnitCallCapError) is read before this PR's outcome rules. feat(standard): retry an attempt that fails on the infrastructure or the agent, with --retries beside --attempts #154's--retriesreplaces the one more run of a rate-limited unit, so the earlierlimits.run_again: falsethat switched it off is gone; main counts a provider's 402 as a usage limit, and the tests follow. The first and fourth commits gave a definition pins of its serving providers, defaults of Served by after review; the last commit removes them, as the reviewer asked, since no benchmark declares one: the diff over main has none, and Served by is the user's alone, as in feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124.