Skip to content

feat(catalogue): add OSWorld2 with OSWorld-V2's own agents, its services on the machine and a null control - #134

Merged
aitianshi merged 26 commits into
ThinkFlowLab:mainfrom
sirotaoleg:feat/osworld2
Nov 19, 2026
Merged

aitianshi merged 26 commits into
ThinkFlowLab:mainfrom
sirotaoleg:feat/osworld2

Conversation

@sirotaoleg

@sirotaoleg sirotaoleg commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #133: this pull request's own commits are the last sixteen. Until #133 is merged, the diff shows its files too. It can be merged after #133.

What

OSWorld-V2's 108 tasks become a bundled benchmark, osworld2, on the Nullius standard, at the upstream's release osworld-v2.1. Each task is a unit in an Ubuntu VM of its own on the upstream's docker provider, set up by the task's own class and scored from 0 to 1 by its own evaluate(). The tasks and their assets are gated on Hugging Face. Nullius does not fetch them, and the entry is gated (#133): a unit's failure from a model endpoint says its status alone, and the driver names an upstream exception by a known class's name at most, never by its words. The Original setup runs one of OSWorld-V2's own agents, M3 or Muse Spark, chosen by the Agent option, from its published script, with every model call sent to the relay. Claude Sonnet 4.6 is the LLM grader of 21 tasks and the simulated user of 6. No provider is pinned: Served by is the user's, as for OSWorld1. Only M3's runs count as comparable with the paper's row. Its Review row and its generated page name the setup "OSWorld-V2 agents". A null control runs one DONE action, then the task's evaluate() on the untouched VM. The entry declares agent.computer/v1. No agent from the Agents page runs it yet. Its texts are one Good to know line and 9 one-line limitations.

The mock websites of 40 tasks are OSWorld-web at the same tag. The guide's steps build them from source and serve them on the machine the units run on. A unit that needs them is refused before its VM starts when they do not answer. For 026 and 041, Nullius makes GitLab on the run's machine before the first of them, with a root token it keeps there, then starts it anew from its snapshot for each, one task at a time; Overleaf's 2 tasks get Overleaf started anew. Each of the 8 tasks that ask for a proxy gets a proxy of its own. The options are the agent, by name, Thinking for M3, and disk write limits for each VM. The website host suffix is asked on the Dataset step, for the four datasets with website or GitLab tasks. Steps per task is with the limits, each agent's own by default, as its wrapper runs it: 100 for M3, 500 for Muse Spark. The VM provider, Docker alone, is on the Where step. By default M3 takes 100 steps with thinking off, as the upstream's M3 script does; the paper ran M3 at 500 steps with thinking on. Muse Spark runs as published at this release, while its published score came from newer upstream code; a limitation says it does not compare, and a docs page says what differs. There are six datasets: All tasks (108, the upstream's test_v2.json), Offline (82), and four by the services their tasks need. Tasks 103 and 104 are not supported: their evaluate() needs FreeCAD on the driver's machine. Review and the run's start ask for Docker and for the scoring programs (ImageMagick, ffmpeg, tesseract) on the driver's machine. As for OSWorld1, the Where step greys out the container card under the Original, whose docker provider needs Docker on the machine. A guide, docs/benchmarks/osworld2-machine.md, prepares a machine, runs the null control on one task as its last setup step, and says what a run takes in time and money. A second page, docs/benchmarks/osworld2-results.md, says how a run differs from the paper and the upstream, and how a unit is scored, stopped or left unscored.

Touches

standard (an option's example and a machine setup's label: the definition check, the benchmark and agent pages, the builder, Review) · catalogue · templates and static (the mark) · docs and site (docs/benchmarks/, the generated pages, AGENTS.md) · routes (tests only) · packaging (the entry's .yml, .Dockerfile and startup/*.py as package data in pyproject.toml)

Before and after

Before: The Benchmarks page before this change

Benchmarks page before

After: OSWorld2 listed, with its mark

Benchmarks page after

Before: The builder's Benchmark step under Computer use: OSWorld1, beside the selected GAIA

Benchmark step before

After: OSWorld1 and OSWorld2

Benchmark step after

Before: The Agents page: "n/7 benchmarks it runs"

Agents page before

After: "n/8": OSWorld2 counts, and no agent runs it

Agents page after

Before: An agent's page, Codex: the benchmarks it runs

Codex's page before

After: OSWorld2 listed: it needs agent.computer/v1, which Codex does not offer

Codex's page after

New: OSWorld2's page: its setup and its proofs by dataset

OSWorld2's page

New: Its options: the agent by name, steps, thinking, the VM provider, the website host suffix and the disk write limits

OSWorld2's options

New: Its dataset and its Good to know line

OSWorld2's dataset

New: Its limitations

OSWorld2's limitations

New: Dataset step: six datasets with their controls; All tasks chosen asks for the website host suffix, and for no token

Dataset step

New: Agent step: OSWorld-V2 agents; the Agents page's agents lack computer use; the settings every agent shares, the disk write limits

Agent step

New: The Original's own settings: MiniMax M3 or Muse Spark by name, and thinking

Agent step, the Original's settings

New: Muse Spark chosen: Thinking, an M3 setting, is gone

Agent step with Muse Spark

New: Model step, Like the paper: MiniMax-M3 on OpenRouter, Claude Sonnet 4.6 as the judge, each served by any provider until one is named

Model step

New: Limits step: Steps per task, each agent's own by default, 100 for MiniMax M3

Limits step

New: Where step: the VM provider, Docker alone; no container or Boat backend here; Docker and the scoring programs checked on this machine

Where step

New: Review: every decision, both prerequisites ready, no model key in this fresh profile

Review step

New: The null control's merged job on test_v2, its first rows

Null control job

New: 052 by the test stand's model in M3: 1.0

Job of 052

Taken at 1400 px, light theme, on a fresh profile. Cut to part of their page: the Benchmark step and Agents pairs (the top 1000 px), Codex's page (the top 1100 px), OSWorld2's page (four parts) and the null control's job (its first rows). The two jobs are runs named under Verified, opened in the app. The Model step, Review, OSWorld2's options, dataset and limitations, and the Dataset, Agent and Where steps were taken again after main's Served by (#124), Boat (#150) and time limits (#191), the shorter texts and GitLab made by Nullius, and checked at 273cca5e: the other pictures differ there only in the sidebar, which they leave out. The two jobs were taken again at c4427915, on main at 0d83e393, whose job page names the judge's button anew (#196); the other pages and steps pictured are the same there. OSWorld2's options and limitations and the Dataset, Agent, Model, Limits, Where and Review steps were taken again at 9526a5b1, after the agents' names, the steps as a limit, the VM provider on Where, the website host suffix on Dataset and Served by with no default.

Verified

  • At 9526a5b1, on feat(standard): let a benchmark keep its tasks gated, pin its own providers, set its own limits and prove a merged null control #133 (e768c632) on main at db6e8f96, on Python 3.12: backend suite (python3 app/tests/parallel.py), 2688 tests in 144 modules, OK; route suite with Node on PATH, 643 tests in 53 modules, OK; the wheel's check (app/packaging/verify.py), OK.
  • OSWorld2's 15 test modules, 375 tests, OK; they need the entry, so none passes without it. 10 are skipped here: they run only when the environment names the upstream with its gated task classes, OSWorld-web or GitLab. Three route test modules and three backend test modules changed for the new entry: the benchmark lists and counts, the Where step's container card, Settings' custom keys, and OSWorld2's case of gated tasks (test_standard_gated.py). Option examples and setup labels change test_standard_options.py, test_prerequisites.py and test_standard_bench_options.py, and the route modules test_catalogue_routes.py, test_agents.py and test_builder_where.py.
  • At 9526a5b1, in a headless Chromium, light and dark, 1400 and 390 px: the builder's Agent step (the agents by name, Steps no longer there), Model step (Any of your choice: no provider filled in), Limits step (Steps per task, 100 for M3 and 500 for Muse Spark), Where step (VM provider docker, fixed) and Review; no page error.
  • In a headless Chromium, light and dark, 1400 and 390 px: the Benchmarks page, OSWorld2's page, the Agents page, Settings and the builder, no page error or cut text. OSWorld2's limitations fit one screen at 390 px, light and dark. /ui-sweep --only osworld2 at fc0bdb1f, this pull request's third commit before the rebases, dropped since (it set limits.run_again, which main's --retries replaced): 6 screenshots, none with a page error or cut text.
  • PYTHONPATH=app python3 -m nullius standard check: ok. python3 app/tests/catalogue_docs.py --write, then the check: up to date. python3 app/tests/field_contract.py: ok. app/site/readme_media.py --check: current. The docs check reports the same two findings in AGENTS.md as on main.
  • Where the runs below ran. The commits named here are of the port's working branches, not in this repository:
    • c3637ffd is this pull request's tree with main's two later commits merged in;
    • f74d1a00 and 58c2b586 add one test commit: two JSON files, no code. It gives OSWorld2's Original one more model, MiniMax M3.1 Flash, the name it is served as, its pin to MiniMax and --max_tokens 32768. That model is a test stand, not in this pull request;
    • 58c2b586 is this entry on the core before feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124;
    • 34e35370 and 8e2e381b are of the port before its move to this repository's history.
  • The controls, recorded in standard.json's proven:
    • Offline, no services, on 2026-11-04, on an earlier commit: the 36 ids are the same twice, and the null control scores 0 on 4 tasks;
    • All tasks, on 2026-11-15, on fc0bdb1f, this pull request's third commit before the rebases, dropped since (it set limits.run_again, which main's --retries replaced); the next commit only writes this record, and the one after changes three tests. The null control ran on all 108 tasks, one launch with each task a shard of its own, at 16 MB/s and 160 IOPS a VM, in six lanes with at most seven VMs at a time; it took 1 h 32 min. 108 and 093 failed their setup under that load and passed alone at 24 MB/s and 240 IOPS, in 24 more minutes. The grader cost $0.14. nullius standard prove osworld2 --dataset test_v2 --job nop=<merged job> passed: the 108 ids are the same twice, 0 of 108 at full score, a mean of 0.0021 against a bound of at most 0.01. Five tasks score above 0 on the untouched VM, as a limitation says;
    • every task is in line with the same control on all 108 tasks on 34e35370, which had the same mean; 077's LLM grader gave 0.054 both times.
  • GitLab made by Nullius, on a server whose GitLab had been set up by hand: the null control of 026, alone, made GitLab first (cloned, configured, its root token, its runner, its snapshot) and scored 0.0 in 15 min; 041, alone, started from that snapshot and scored 0.0 in 7 min; a snapshot of an older stamp was made anew by itself. No model call, no GitLab left after either, and the token in no file of the jobs.
  • An agent's score on every kind of task, each above the same task's null control (0.0 in each):
    • Seven kinds, on 58c2b586, by the test stand's model in OSWorld-V2's M3 agent, from the command line:

      • desktop apps, 045: 1.0;
      • mock websites, 052: 1.0;
      • Overleaf, 072: 0.775;
      • GitLab, 026: 0.5, a partial score;
      • a proxy, 075: 1.0 (recorded 0.9999999999999999), with the unit's proxy carrying new outbound connections while the agent acted;
      • the public internet, 027: 1.0;
      • the LLM grader, 003: 1.0, with both grader calls answered 200.

      Each of the seven gives its recorded score again at c3637ffd when its outcome is computed from its saved records, with no VM and no evaluate().

    • The eighth, the simulated user: 007, 0.8, by Muse Spark on 8e2e381b; the simulated user answered 7 times. It was not run on this base.

  • Live on this base, against OpenRouter:
    • on c3637ffd, the null control on 087, a graded task: 0.0. The grader's call went to Anthropic by the definition's pin, and standard.served reads held: true;
    • on f74d1a00, the test stand's model on 052: 1.0. All 68 calls went to MiniMax by the test commit's pin and named it, held: true, with no error.
  • This pull request's own agents have no positive score on this base. M3 ran one task here, 017, on the core before feat(standard): choose the serving providers that may answer a model, in the builder and on the command line #124 (90fde73e), and scored 0.0: the agent declared the task failed at its 120th step. Muse Spark ran none. Their scores before the move, on 34e35370 and earlier:
    • M3 on 9 tasks: 070 1.0 twice, 030 1.0, 045 1.0, 013 1.0 then 0.0, 031 0.714, 017 0.429, 010 0.0 three times and once an environment error; 026 and 098 left no score (a full context window, a spend guard);
    • Muse Spark on 3 tasks: 045 1.0, 007 0.8; 052 left no score (a launch error of ours, then a spend guard).
  • Every attempt before the move, two test stands' models included, was recorded and triaged: 61 attempts on 33 tasks. 43 were scored (21 full, 11 partial, 11 zero), 10 ended in an environment error and 8 were stopped by a guard or by hand. None was an agent error. Each of their full and partial scores was checked for the false-success patterns the port's audits found: none.
  • Not run: a positive score of M3 or Muse Spark on this base; the simulated user on this base; task 041 with an agent; a parity run of the paper's 108 tasks; the 10 skipped tests with the upstream named; the docker tier; the site build.

Risk

  • OSWorld2's driver and OSWorld-V2's scripts run only when someone runs OSWorld2. The Agents page counts eight agent benchmarks, and the generated agent pages gain an OSWorld2 column. No agent there declares agent.computer/v1.
  • Scores. A run compares with the paper's row by its model alone: Docker instead of AWS, no US residential proxy, release osworld-v2.1, the grader through OpenRouter, and by default 100 steps with thinking off. A wrong error kind would move a unit between "environment error" (left out of the adjusted rate) and the agent's error (counted).
  • Steps. A Muse Spark run that sets no step limit now takes up to 500 steps, its wrapper's budget, where it took 100: a longer and dearer episode. M3 keeps 100. A saved recipe or command that names a number keeps it.
  • The machine.
    • VMs, the mock websites, GitLab and the proxies run on the machine the units run on, outside a run's constraints in Settings; -j bounds them.
    • GitLab's Docker in Docker and the websites' image build run as privileged containers.
    • In each of the driver's processes, startup/sitecustomize.py wraps the creation of the VM's container: its network, its ports on 127.0.0.1, its disk and its write limits. The VMs need a free Docker subnet.
    • The VM's clock is UTC+8: a dated task set up before 16:00 UTC and acted on after can score 0.
    • The first unit of 026 or 041 on a machine clones Task-Web/gitlab at its pin, pulls its images (about 7 GB) and makes its snapshot: about 8 minutes more, once.
  • Secrets. GitLab's root token is made on the run's machine and kept there in a file for its user alone, beside GitLab's clone; only the processes of 026 and 041 get it. Model keys stay with the relay. Tasks 001, 006 and 010 start their mail client offline, unlike the upstream.
  • Stacked on feat(standard): let a benchmark keep its tasks gated, pin its own providers, set its own limits and prove a merged null control #133. Once it is merged, this branch is rebased onto main.
  • Undone by reverting the commits; the catalogue entry goes with them.

harenome added a commit that referenced this pull request Nov 16, 2026
Closes #136

## What

On Runs → By launch, an agent or control with 2 or more shards is now
one row, when `cli merge` would merge those shards. The row shows the
numbers of the merged job. `launches.fold` makes that job in memory: the
trials of every shard, and the job record from `merge.result_of`.
`result_of` is the code that `merge` used inline to compute the job
record, now a shared function. `merge.check` keeps the trial records
that it already reads.

- The rate is over every trial with a verdict, never an average of shard
rates.
- Units that no shard ran come from `merge.check` and stand beside the
count ("2 of 8 not run"). They are never misses. The notes of `merge`
are the tooltip.
- The state is the most telling state of the shards, with a warning such
as "1 of 2 shards stopped".
- The header gives the dataset size, not the size of one shard.

The row is folded by default. A click on the row, or Enter or Space on
its chevron, unfolds one row per shard ("shard 2 · host-2"), each with
its Open. The fold uses the shared chevron and slide of
`disclosures.js`, and respects reduced motion. A refresh of the list
keeps an unfolded row unfolded. Controls fold the same way, stay
unranked and stay out of Compare. The task-check line is judged over
every shard: "Task checks incomplete: oracle not run on shard 2."

A participant with one shard keeps its plain row. Shards that `merge`
refuses also keep their rows. Job pages, `cli merge` and merged jobs do
not change. The header counts every shard ("8 runs"), every shard has
its row, and each folded row says how many shards it holds ("2 shards").

A fold reads every trial record of its shards. It is kept in memory for
the shard jobs that the workspace cache gives, and it is read again only
when the workspace parses a shard again. A new set of shards replaces
each kept fold that shares a shard with it, so the cache holds one fold
per shard at most.

## Touches

backend · the standard (`merge.py`) · routes · templates and static ·
docs and site

## Before and after

One launch over 2 machines, 8 units: 2 agents, nop and oracle. Shard 2
stopped while terminus-2 ran (2 of its 4 units), and the oracle did not
run on shard 2. Light theme, 1280 px.

| Before | After, folded |
| --- | --- |
| <img width="980" height="670" alt="1-before-shards-apart"
src="https://github.com/user-attachments/assets/990ceda8-8212-4003-ab1c-f380be783fc9"
/> | <img width="980" height="421" alt="2-after-folded"
src="https://github.com/user-attachments/assets/f37d9af1-7b3b-4b05-abc6-7dca2da1c983"
/> |

Before: 8 rows, 2 rates for each agent, "4 tasks", "oracle not run".
After: 4 rows, "8 tasks". terminus-2 shows 50% and "3/6 · 2 of 8 not
run" with "1 of 2 shards stopped". The task-check line reads "Task
checks incomplete: oracle not run on shard 2."

| After, unfolded (click and keyboard) | After, dark, 390 px |
| --- | --- |
| <img width="980" height="595" alt="3-after-unfolded"
src="https://github.com/user-attachments/assets/b30f3189-d737-4658-8701-a3bbc434ad4a"
/> | <img width="358" height="844" alt="4-after-dark-390"
src="https://github.com/user-attachments/assets/6c4b106c-218b-4308-a934-ecac1f96934d"
/> |

## Verified

- Backend test `Fold.test_a_fold_reads_as_the_merged_job`: three shards,
one absent, and one with a planned unit that did not run. The folded job
equals `parse_job` of the `merge.merge` output on every field the row
shows (solved, rates, outcomes, planned, errors, start, end, time,
tokens, cost, mean score, budget). The rate is not the mean of the shard
rates. The units not run and the notes are those of `merge`. Shards that
`merge` refuses are not folded.
- Backend test
`Fold.test_a_fold_is_read_again_only_when_the_workspace_parses_a_shard_again`:
a second fold of the same cached jobs does not call `merge.check`. After
a trial record changes, it calls `merge.check` again. After a third
shard appears, the cache holds only the fold of the three.
- Route tests `test_launch_shards`: 8 shards give "8 runs", 8 shard rows
and 4 folded rows that each say "2 shards". The rows are folded by
default. Each shard row has its Open, and the folded row has none. The
rate and time of the folded row equal those of the merged job. "2 of 8
not run" and "1 of 2 shards stopped" show. The controls fold and stay
out of Compare. A nop that passes one task on shard 2 gives "A control
did not behave". A stopped nop shard gives "nop stopped on shard 2".
Participants with one shard keep their plain rows.
- On `c7af06d5`: backend suite `python3 app/tests/parallel.py` (1989
tests) and route suite `python3 app/tests/parallel.py app/ui_tests` (328
tests) pass in a clean container.
- Browser, with the fixture above: folded and unfolded, light and dark,
1280 and 390 px. No page errors and no horizontal page scroll. A click
on the row unfolds it, and the shard rows slide open and closed. Tab
moves the focus to the chevron of the oracle, Enter unfolds it, and
Space folds it again. With reduced motion, the rows show at once. At 390
px, a long agent name wraps and the chevron stays on the line of the
name. The name, its "2 shards" line and the shard rows start at the same
left edge.
- Render time of `/runs?view=launch`, median of 15 requests after a
first one, with the same parsed jobs given to the page. 2 agents and nop
and oracle, each 4 shards of 500 units (8000 trials): 71 ms before this
change, 101 ms after. The first render reads the trial records once: 540
ms, against 199 ms before. The rest of the added time is the outcome
bars of the 4 folded rows, over 2000 trials each. 8 shards of 1 unit
each (32 trials): 11 ms before, 13 ms after.

## Risk

If the fold is wrong, a row shows numbers that differ from the merged
job. The backend test compares the two directly. `merge` writes the same
`result.json` as before through `result_of`, and its tests pass
unchanged. The first render of a large sharded launch reads all its
trial records once (+0.3 s for 8000 trials). Later renders use the cache
until a shard changes. The cache holds one fold per shard at most, with
the parsed jobs that the workspace cache also holds. Open pull requests
#124, #133 and #134 also change `merge.py`: the second to merge must
rebase. Revert the commit to undo.
harenome added a commit that referenced this pull request Nov 16, 2026
…l everywhere (#148)

Closes #147

## What

Each model provider is now one record that every feature reads. A
feature asks for the protocol that it speaks, and gets the providers
that serve it.

- **The registry** (`app/nullius/providers.py`). `PROTOCOLS` defines
each wire format once: its default path and how the key is sent (`chat`,
`responses`, `messages`, `systemone`, `decisions`, `openai-decisions`).
`PRESETS` holds OpenRouter, OpenAI, Anthropic and TypeSafe. `record()`
returns `{name, base, key, endpoints}` for a preset or a custom
provider. `key` is the name of the secret setting, never the key.
- **Storage** (`app/nullius/settings.py`). A record is one JSON setting
of a new kind, `provider`. A custom provider is `MODEL_<ID>`. The
endpoints added to a preset are `OPENROUTER`, `OPENAI`, `ANTHROPIC` or
`TYPESAFE`. For these settings the environment does not replace the
file. Its value is a partial object, merged over the file's value (RFC
7396, `null` removes a field). Settings shows read-only only the fields
that the environment sets. Keys keep their names and stay separate
secrets.
- **Migration.** `MODEL_<ID>_NAME` and `MODEL_<ID>_API_URL` are still
read as `name` and `base`. Saving a card moves them into the record.
`nullius migrate --providers` previews that rewrite of `settings.json`,
and `--apply` writes it. Recipes do not change. `local` and `local:<ID>`
still read as `custom`.
- **Consumers.** The registry is read by the builder, the provider test,
onboarding, `execution`, standard runs, the relay, the CLI's
`--provider` and both judges. The builder's JavaScript copy of the list
is gone. The builder offers a provider only where it serves what the
recipe's agents and judge speak, so Anthropic appears for an agent of
the standard that speaks `messages`. Review, Run (`web/execute.py`) and
`run.Run` refuse a provider that does not fit. The relay sends
`x-api-key` and `anthropic-version` on a `messages` route only, and a
bearer header on a benchmark's own routes.
- **Settings cards.** A card shows Base and a collapsed Endpoints
section with one row for each protocol. The placeholder is the default
URL. A base on a preset's host suggests that preset's endpoints. A
preset's known endpoints are fixed. An endpoint added to a preset is a
path, or a full URL on the preset's own host, because the preset's key
goes there.
- **System One** (`app/nullius/judge/client.py`) asks each decision
protocol: the TypeSafe shape on `systemone` and `decisions`, and its own
adapter for `openai-decisions`. It keeps the provider's `usage.cost` and
records a `refusal` as no answer. A 402 says what it means. A judge
never follows a redirect with the key.
- **Judge cards.** Each judge has Provider, then Protocol when the
provider serves two protocols of that judge (else one quiet line, "via
..."), then Model. Each judge's provider is a select whatever the count,
because the list is the user's own providers and grows or shrinks with
them; cards are kept for fixed sets of choices. New settings:
`LLM_JUDGE_PROTOCOL`, `SYSTEM_ONE_PROTOCOL`. On OpenRouter, System One
offers OpenRouter's public list of decision models, asked without a key,
cached 6 hours, with the fixed list as the fallback.

The seven commits are atomic and in reading order: registry and storage,
consumers, Settings cards, decision protocols, judge cards, tests, docs.

## Touches

backend (`providers`, `settings`, `migrate`, `onboarding`, `execution`,
`judge/client`, `recipes/launch`) · the standard (`run`, `relay`,
`recipe`, `cli`, `definitions`, `fidelity`) · routes (`web/settings`,
`web/execute`, `web/filters`) · templates and static (Settings cards,
judge cards, builder core) · docs (`docs/settings/model-providers.md`,
`system-one.md`, `llm-judge.md`, `README.md`, `agent-keys.md`,
`docs/builder.md`, `docs/judgement.md`, `docs/standard.md`,
`docs/cli/README.md`, `AGENTS.md`, `app/.env.example`)

## Before and after

Settings at 1280 px, light theme. Fixture keys only.

Model providers. TypeSafe is a preset like the others, so it comes after
the run providers.

| Before | After |
| --- | --- |
| <img width="748" height="431" alt="1-before-providers"
src="https://github.com/user-attachments/assets/1dd2338e-0b32-4631-8895-a6df2ba7fc29"
/> | <img width="748" height="431" alt="2-after-providers"
src="https://github.com/user-attachments/assets/ff3bb210-22ef-4bff-bbe4-df4ac4488519"
/> |

A preset card (OpenRouter), Endpoints open. Known endpoints are fixed.
Other protocols are editable rows.

| Before | After |
| --- | --- |
| <img width="748" height="360" alt="3-before-preset-card"
src="https://github.com/user-attachments/assets/5fb776e9-1c86-4cf0-8508-24041b6f9042"
/> | <img width="748" height="706" alt="4-after-preset-card"
src="https://github.com/user-attachments/assets/9058e8e4-339e-49da-8a44-67b2ffcdf8bf"
/> |

A custom provider, Endpoints open, with one endpoint set.

| Before | After |
| --- | --- |
| <img width="748" height="451" alt="5-before-custom-card"
src="https://github.com/user-attachments/assets/c725f642-3d22-4b8d-8fb0-c702e415594d"
/> | <img width="748" height="897" alt="6-after-custom-card"
src="https://github.com/user-attachments/assets/afcbcb01-fed9-4ef3-87c2-883d73594edc"
/> |

Judges. System One on OpenRouter serves two of its protocols, so it
shows Protocol. The LLM judge on OpenAI serves one, so it shows "via
OpenAI Chat Completions".

| Before | After |
| --- | --- |
| <img width="748" height="504" alt="7-before-judges"
src="https://github.com/user-attachments/assets/a089a1bd-9f72-4ca6-bff1-e1a652a83ec1"
/> | <img width="748" height="503" alt="8-after-judges"
src="https://github.com/user-attachments/assets/61ef9ae3-e080-4e54-99a1-dc43d4960ba6"
/> |

The custom card at 390 px:

<img width="358" height="1153" alt="9-after-custom-card-390"
src="https://github.com/user-attachments/assets/dce35d02-44b1-4ccb-8362-c84129e1eb87"
/>

## Verified

The branch is based on `main` at 1cc37ac. I tested its tip, each suite
in a new rootless Podman container (`python:3.12`, `--init`, uv on PATH,
Node for the route suite):

- Backend suite (`python3 app/tests/parallel.py`): 2029 tests in 118
modules, OK. `main` at 1cc37ac, in the same container: 1992 tests in
116 modules, OK.
- Route suite (`python3 app/tests/parallel.py app/ui_tests`): 441 tests
in 42 modules, OK. `main` at 1cc37ac: 431 tests in 41 modules, OK.
- New modules: `test_provider_registry` (17 tests: the merge, `null`,
partial environment values, the old settings, record checks, the preset
host rule, keys never in a record), `test_judge_decisions` (18 tests:
each decision protocol's request and reply, `usage.cost`, `refusal`,
402, no redirect, the cached model list and its fallback),
`ui_tests.test_judge_cards` (2 tests: the protocol row and the live
list). Existing modules gained tests for the relay's headers, the
builder's provider filter, Harbor recipes on Anthropic and the Settings
cards.
- End to end, against the server (`app/app.py`) and the CLI, in a
container with no network, with a new `NULLIUS_HOME` for each case and
fixture keys only. 37 checks, all pass:
  - No settings: the four presets and the OpenAI-compatible card.
- A custom provider in `settings.json`: its card, its merged record, the
default path for a protocol without an entry, System One offers it for
its `decisions` endpoint, no key in the page.
- `MODEL_GW='{"base": "https://gw-eu.example/v1", "endpoints":
{"decisions": null}}'` in the environment: the base changes, the name
stays the file's, the endpoint goes, System One no longer offers it. The
card shows Base read-only and Name editable. The file does not change.
- `nullius migrate --providers`: the preview names the moves, prints no
value and leaves the file as it was. `--apply` writes the record,
removes the old lines, keeps the key and the file mode 0600. A second
run finds nothing to move.
- Old variables in the environment (`MODEL_API_URL`, `MODEL_<ID>_NAME`,
`MODEL_<ID>_API_URL`, `LLM_JUDGE_PROVIDER=local:<ID>`): read as before.
- System One's model list on OpenRouter, offline: the route answers with
the built-in list in less than the 5 s timeout, or with a cached list,
however old.
- Browser (Chromium, Playwright), on this branch and on `main`, at 1280
px and 390 px, light and dark, with no network: the provider list, a
preset card and a custom card with Endpoints open, and both judge cards.
No page errors and no horizontal scroll.

Not verified: live calls to OpenRouter Decisions and OpenAI Decisions,
and a whole run on Anthropic. The tests use fixtures.

## Risk

A wrong record or header can send a call to the wrong URL, or with the
wrong key header. Undo with a revert. The old settings are still read,
so a revert keeps working with them. Records written by a card stay in
`settings.json` and the older code ignores them.

- **Migration on edit.** Saving a custom provider's card moves
`MODEL_<ID>_NAME` and `MODEL_<ID>_API_URL` into `MODEL_<ID>`. After a
revert, that provider loses its name and base until they are set again.
`nullius migrate --providers` without `--apply` shows what would move.
- **New reserved names.** `OPENROUTER`, `OPENAI`, `ANTHROPIC`,
`TYPESAFE` and `MODEL_<ID>` are now settings. An agent key listed under
one of these names is left out, and Settings says so. A shell variable
such as `MODEL_PATH` that is not a JSON object is not read as a record.
- **`--provider anthropic` on the CLI.** The key header now follows the
agent's protocol. An agent that speaks only `chat` is refused on
Anthropic, where it used to get `x-api-key` on OpenAI-style paths.
- **Runs and custom endpoint paths.** The relay forwards the agent's own
path below the provider's base. An Endpoints row changes where the
judges and the test ask, not where a run's calls go.
- **`openai-decisions` field names** come from OpenAI's documentation,
not from a live reply.
- **Open PRs on the same files.** This change does not depend on any of
them. Whichever merges second rebases. A test merge of this branch with
each one:
- #124 (OpenRouter serving providers, `--served-by`): conflicts in
`providers.py`, `standard/run.py`, `standard/relay.py` and
`docs/settings/model-providers.md`. Each conflict is small. Both add
functions at the same place in `providers.py`. This branch removes
`run.PROVIDERS`, where #124 adds `routing` to the OpenRouter entry: that
flag becomes a field of the OpenRouter preset. Both change
`Relay._headers`.
- #133 (stacked on #124): the same four files, and also
`standard/fidelity.py` (one `key_header=` line, now `r.key_header`) and
`docs/standard.md` (one table row each).
- #134 (stacked on #133): the same as #133, and also
`ui_tests/test_settings_page.py` (one placeholder assertion, which both
change).
- #124, #133 and #134 also conflict with `main` alone in
`standard/merge.py` (and #133, #134 in `tests/test_standard_shard.py`),
which this branch does not touch.

@aitianshi aitianshi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GitLab for tasks 026 and 041 should be supported automatically, or left out of this PR.

Right now it is half done. Each unit starts GitLab from its snapshot and removes it afterwards, which is automatic. The one-time preparation is not: docs/benchmarks/osworld2-machine.md step 9 asks the user to clone Task-Web/gitlab at the pinned commit, run gitlab_server.py configure, compose pull and init, choose a root token, paste it into Custom keys as GITLAB_PRIVATE_TOKEN, and set GitLab host. A user should not have to set up a GitLab server by hand to run a dataset.

The builder makes this visible on every card. It shows Add GITLAB_PRIVATE_TOKEN in Custom keys (some tasks may need it). even on "Offline, no services", whose 36 tasks never touch GitLab. That happens because no dataset names an environment, so credentialWarnings treats the gitlab environment's secret as possibly needed everywhere.

Please choose one of these:

  1. Support it automatically. Nullius prepares GitLab on the run machine as part of the run's prerequisites (nullius/prerequisites.py): clone at the pin, configure, pull, init. It generates the root token there and keeps it on that machine, so the user never types or pastes one. The builder then shows nothing about GitLab unless the preparation fails.
  2. Remove it from this PR. Drop the GitLab parts (gitlab_server.py, gitlab.compose.yml, the gitlab environment and option, the licence, step 9 of the doc, its tests). Tasks 026 and 041 then stay out of the offered datasets until a follow-up PR supports them fully.

Either way, a selection that cannot need the token should not warn about it.

"good_to_know": [
"Scores are partial credit: each task's own evaluate() gives 0 to 1, and the upstream's score is the mean."
],
"limitations": [

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Limitations are far too long. Please cut them down a lot.

There are 25 items and about 2,200 words; one item alone (the LLM grader) is 332 words. On OSWorld2's page they fill several screens, and nobody will read them before choosing a run.

The UI rules (.claude/rules/ui.md) say Good to know and limitations must change a user's choice, the action they have to take, or how they read the results. They also say to leave out dependency and installation details that Nullius handles itself, and that removing text beats adding it. Most of these items are how the driver works inside, how errors are classified, or how the machine is set up:

  • installation and machine setup: mock websites build, GitLab, proxy image, VM resources, write limits, Settings constraints, output kept per trial. These belong in docs/benchmarks/osworld2-machine.md, or nowhere if Nullius does them;
  • outcome mechanics: how a setup failure, a timeout, the upstream's own failures, the 10-hour stop, a grader refusal or a context overflow is scored. These belong in the docs or in the trial's own record.

What a user actually needs is roughly this, one short sentence each:

  • not the paper's setup: another release, no residential proxy, and 100 steps with thinking off by default (Steps 500 and Thinking adaptive restore the budget);
  • Muse Spark's published score came from newer upstream code and does not compare;
  • the grader and the simulated user are Claude Sonnet 4.6 on OpenRouter: with another provider those tasks fail;
  • tasks that use the public internet can change outcome with the live sites;
  • tasks 103 and 104 are not supported;
  • a slower machine can lower scores;
  • the null control scores above 0 on five tasks.

Aim for under 10 items, each one line.

@aitianshi

Copy link
Copy Markdown
Collaborator

The screenshots in the description can't be read without opening them, and opening them fails.

Each image is shown at width="420", so the text in it (the limitations, the options) is too small to read and the reader has to click it. Clicking it opens the private attachment link. For a reviewer here that gave an empty file: "The file 707192284-87ab46cf-...png could not be opened because it is empty" (the "OSWorld2's limitations" screenshot).

Please put the screenshots directly in the description at full size, for example by dragging them into the editor and leaving GitHub's own ![...](...) line without width. Crop each one to the part that matters, so it can be read on the page without clicking a link.

"websites": {
"label": "Website host suffix",
"summary": "WEBSITE_HOST_SUFFIX: none, or the host suffix OSWorld-web's sites are served under for this machine's VMs, a wildcard DNS name of the address the stack listens on, the gateway of the VMs' own network (10.213.0.1.nip.io). The 40 tasks that use the sites need it.",
"pattern": "none|(?=.{3,200}$)[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?(?:\\.[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?)+",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The page should show an example for these options, not the regex.

OSWorld2's page prints the raw pattern of all five free-text options (Website host suffix, GitLab host, GitLab write share, Disk write MB/s, Disk write IOPS). It shows none|(?=.{3,200}$)[a-z0-9](?:[a-z0-9-]{0,61}... where a user needs to see what to type. catalogue/benchmark.html only hides the pattern for temperature ("Custom decimal, e.g. 0.5"); every other pattern option falls through to the raw regex. Before this PR no other option had a pattern, so nobody saw it.

Suggested:

  • each pattern option carries an example, e.g. "example": "10.213.0.1.nip.io", "gitlab.10.213.0.1.nip.io", "16:160", "16", "160";
  • the page shows "e.g. 16:160" instead of the pattern, the same way temperature already does, so the special case goes away;
  • the pattern stays for validation only;
  • the summaries drop the internal names they start with (WEBSITE_HOST_SUFFIX:, GITLAB_URL:, device_write_bps), following the "Names, not keys" rule in .claude/rules/ui.md.

},
"websites": {
"label": "Website host suffix",
"summary": "WEBSITE_HOST_SUFFIX: none, or the host suffix OSWorld-web's sites are served under for this machine's VMs, a wildcard DNS name of the address the stack listens on, the gateway of the VMs' own network (10.213.0.1.nip.io). The 40 tasks that use the sites need it.",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please make the option and setup descriptions much shorter.

On the page, each option's description runs 2 to 4 lines (35 to 44 words for Website host suffix, GitLab host, GitLab write share and Disk write MB/s). The two Setup blocks are longer still (Docker is 54 words). They explain how things work inside: Caddy, snapshots, device_write_bps, 1024 * 1024 bytes, the network nullius-osworld2. A user only needs to know what the value does and when to set it. The UI rules (.claude/rules/ui.md): "Less text", "Removing text beats adding it", and the details go on the entry's own docs page.

For example:

  • Website host suffix: "Where the mock websites are served. Needed by the 40 website tasks."
  • GitLab host: "Where GitLab is served. Needed by tasks 026 and 041."
  • GitLab write share: "Disk write limit for GitLab, as MB/s:IOPS."
  • Disk write MB/s / IOPS: "Disk write limit for each VM."
  • Setup: Docker: "Docker without sudo; KVM recommended."
  • Setup: scoring programs: "ImageMagick, ffmpeg and tesseract, used to score 6 tasks."

Two related things on the same page:

  • the Setup headings show raw keys (Setup: docker, Setup: scoring_programs) because benchmark.html prints the key. They need a label, per "Names, not keys";
  • the thinking option has no label.

@sirotaoleg
sirotaoleg force-pushed the feat/osworld2 branch 2 times, most recently from 273cca5 to c442791 Compare November 18, 2026 02:31
@aitianshi

Copy link
Copy Markdown
Collaborator

The four fields on the Agent step (VM provider, Website host suffix, Disk write MB/s, Disk write IOPS) are not the agent's settings. Please move or remove them.

All four default to none or docker, and a run works without touching any of them. Per .claude/rules/ui.md, "One step, one question; a setting sits with what it changes": the Agent step is who runs it and their settings.

  1. Website host suffix: make the mock websites automatic, then drop the field.

    • It depends on the tasks, not the agent. 40 tasks need the websites (websites.TASKS), and 4 of the 6 datasets contain some. Offline, no services and No hosted services contain none, yet the field shows for them too.
    • The value is always the gateway of the VMs' network, which Nullius creates itself (10.213.0.1.nip.io), so there is nothing for the user to choose.
    • The setup is still manual (machine guide, build and start the websites; proxy image too). Like GitLab now: a unit that needs the websites builds the stack once per machine under an exclusive lock, units then share it (the shared lock already exists) and reset it as today. Two machines, two stacks: the VMs reach the gateway of their own machine's network.
    • The dataset then carries a prerequisite derived from its task list (no hand-written flag per dataset), shown on Review, with nothing to type.
    • If a field has to stay meanwhile, it belongs on the Dataset step, shown only when the chosen dataset contains a website task (when keyed on the dataset).
  2. Disk write MB/s and IOPS: remove them from the benchmark. They are not in the upstream, the paper or any dataset; the standard has a benchmark carry what its upstream offers. A disk limit is a property of the machine. If it is needed, it is a separate change in the machine constraints (constraints.py, Settings) for every benchmark. Please remove the options, the machine guide's "Write limits and load" and the driver code that applies them.

  3. VM provider: keep the option, move it to the Where step. It chooses where the VMs run, the same question as Where. The upstream offers aws, virtualbox, vmware, docker and azure; only docker is wired. While it has one value, show it as fixed, not as a selector, per the rule on a single choice. It becomes a choice when a second provider is wired.

@aitianshi

Copy link
Copy Markdown
Collaborator

Agent and Steps options: show names, and make Steps a limit as OSWorld1 does.

Agent. The cards show the raw keys m3 and muse_spark (OSWorld1's select does the same with its 12 keys), against "Names, not keys" in .claude/rules/ui.md. The readable text is only in about. Proposed:

  • an option may carry names: {value: "Display name"} beside values and about; definitions.py checks each key is one of its values;
  • the builder shows the name on the card or in the select, about stays the tooltip; the recipe still stores the key, so saved recipes and the CLI do not change;
  • OSWorld2: "names": {"m3": "MiniMax M3", "muse_spark": "Muse Spark"}; OSWorld1's agents the same way (PromptAgent, Claude computer use, GPT-5.4, OpenAI CUA, Qwen, OS-Symphony, EvoCUA, AutoGLM, AutoGLM-V, Mano), spelled as each upstream spells them.

Cards for 2 values and a select past 5 stay as they are.

Steps. OSWorld2 offers 100 / 300 / 500 as cards on the Agent step. OSWorld1 has the same setting as a limit: limit: true, a free number, shown on the Limits step as "Steps per task", its default each agent's own through by. Please do the same here:

  • limit: true, a pattern and an example, so it moves to the Limits step;
  • "by": {"agent": {"m3": "100", "muse_spark": "500"}}: each agent's published budget as its default. Today both default to 100, though Muse Spark's published budget is 500;
  • no 300: it is in neither the paper nor the upstream ("no invented numbers").

@aitianshi

Copy link
Copy Markdown
Collaborator

Model step: OSWorld2 should act like OSWorld1, with no provider pinned.

With Any of your choice and minimax/minimax-m3, the step opens with Served by: minimax and, for the judge, Served by: anthropic already filled in. The user has to remove them to use any provider. OSWorld1, which declares no pin, shows "any provider" in both fields.

Why: the definition pins them (setups.original.pin, score.judge.pin), recipe.compile returns them as served_pins, and the builder applies them whatever the model choice (servedPin in command-builder-core.js does not look at model_choice).

Please remove pin from OSWorld2's standard.json, both setups.original.pin and score.judge.pin, so it behaves as OSWorld1 does. That also settles the disagreement with #124 raised on #133: there, Served by is the user's choice, never a default of the definition. If a pin is kept anywhere, it should apply only under Like the paper, never under Any of your choice.

@aitianshi

Copy link
Copy Markdown
Collaborator

A first integration runs the upstream exactly as published: please remove every change to it from this PR.

docs/standard.md: "Never patch its bugs: say it in the limitations; a fix is a fork's, brought in as a version", and a version is a fork's repository at a commit, "never a patch of ours". This PR changes the upstream in several ways:

Change Where
Mail profiles of tasks 001, 006, 010 rewritten: logins and cookies removed, Thunderbird started offline driver.py _assets, _signed_out
The upstream's DataImpulse proxy replaced by a local tinyproxy for 8 tasks proxy.py
Guest disk raised to at least 40 GB (the docker provider gives 32 GB) sitecustomize.py floor_desktop_env
The agent's loop made to answer DONE when a Nullius guard trips sitecustomize.py stop_when_asked
docker-py's create wrapped (timeout, adoption, network, write limits), the provider's LOCK_TIMEOUT changed sitecustomize.py
A grader failure or an unread VM or site turned into an error, where the upstream writes 0 driver.py _unread, _grader_failed

For this entry:

  • run the upstream's code and assets unchanged: no startup/sitecustomize.py, no rewritten assets, no replaced proxy;
  • the driver is glue only: start the upstream's script, route its model calls through the relay, start the services it needs, read its results, clean up;
  • a guard that trips stops the agent's process from outside, never by changing its loop;
  • the score is the one the upstream writes. A grader failure may be noted beside it, not change it;
  • what the upstream does badly goes in the limitations, short, when it matters to a choice or to reading a result (tasks that need a paid proxy can be left out of the datasets).

The fixes are welcome as a version: commit them to a fork of OSWorld-V2, declare it as versions.<name>.upstream = {repository, tag, tested}, prove its own null control, and mark it not comparable with the paper.

…on top of --served-by: gated tasks, a definition's pins as its serving pins, a pinned route's refusal as the unit's, limits of their own, and prove of a merged control

A definition's pin (setups.original.pin, score.judge.pin) joins the run's --served-by pins and wins over
--served-by for its model; ApiProviderRouteError reads the pin and reply.served_by. A failed Responses
object reads reply.end and reply.error.
…st 429

A unit whose driver scored it after a guard's stop keeps that score, marked stopped, though its last own call got a 429.
… own last 429, not a side call's or a context window's stop

The rule reads the agent's own last call, a guard's line in the stop file and a score stopped by other than the context window; a cap's refusal is still the cap's.
…h --served-by replaces, as ThinkFlowLab#124 has the user choose

A model the definition pins (setups.original.pin, score.judge.pin) goes to the
user's providers when --served-by gives it others: the run says so, records the
definition's under standard.pins_replaced, and the Original's run does not
compare with the paper. The recipe's compile gives each field's default
(served_pins) and sends a field's list, its own or the default, with every job
that calls its model; the builder's Served by starts with it, and Review says
when a field's differ. --served-by without MODEL= names the model an Original
sends, and prove --job refuses a control whose pins were replaced.
…r place, the recipe keeping the value; OSWorld1's agents by their upstream's names
…e the work runs on Where, a single value as fixed, and what the datasets need on Dataset; OSWorld1's VM provider on Where
…eclares, with their route errors; Served by is the user's alone, as ThinkFlowLab#124 has it
… upstream's own runners, mock websites, GitLab and proxy, proven on test_v2, and how to prepare a machine for it

OSWorld2 runs each of the 108 tasks of test_v2 in a VM of its own through the upstream's docker provider, with the mock websites its tasks need served on the machine, and GitLab and a proxy started for each unit that needs one. Its Original runs the upstream's agents with their models pinned to their providers on OpenRouter, and its null control is proven on test_v2 at 0 of 108. The tasks are gated, so a failing endpoint is kept as its status alone. Its machine page says what a machine needs, how to check it and what a run costs.
…g programs, and fake the sweep's clock for the sweep alone

Two tests made a Run of OSWorld2, which asks for ImageMagick, ffmpeg and
tesseract on the machine: they now say the programs are there, as the
test of their refusal does. The sweep's tests replaced time.sleep for
every module, so a slow docker's wait in subprocess advanced the fake
clock and the test's own steps: the fake now replaces the sweep's time
module alone.
…he rest to its machine setup and a page on reading its runs
… keep OSWorld2's option and setup lines short and named

An option with a pattern may carry an example its pattern allows besides
its default, checked when given, so an entry from before still loads;
every bundled pattern carries one. Pages, the builder, a refused value
and the generated pages show the example; the pattern only validates. A
default named default reads Provider default for any option.

A machine setup may carry a label, shown in place of its key on the
benchmark page, in the builder, in Review and in the container note.
OSWorld2's option and setup lines say what a value does and when to set
it, without internal names; the machine guide keeps the rest.
…es and datasets, its refusals by the option's name, and its results page to what a trial says

The definition's summaries, dataset texts, option values and the null control's
source say what a user chooses, without script names or code literals; the one
VM provider is fixed. The mock websites' refusals name Website host suffix, and
the driver's say stopped and the judge's calls. Read an OSWorld2 run keeps what
differs from the paper and a table of what a trial says; the machine guide keeps
one rule for write limits.
…root token of its own

Tasks 026 and 041 no longer need a GitLab set up by hand. Their first unit
on a machine clones Task-Web/gitlab at its pin beside the upstream's clone,
configures it for the mock websites' suffix, pulls its images and makes
its snapshot, under the GitLab lock and its warden; later units start
from that snapshot. The snapshot keeps a stamp of the stack, the host and
the token, and is made anew when any of them changes. Nullius makes the
root token there, a file for this user alone beside the stack, and gives
it to the processes of those two tasks only. The run goes where its
driver runs, here or on a host.

The GitLab host option and the GITLAB_PRIVATE_TOKEN licence are gone, so
the builder asks for no token: GitLab's host is gitlab. before the
websites option, whose Caddy serves it. Step 9 of the machine guide goes.
…esh, and ask nothing about GitLab

The GitLab write share option goes: a GitLab unit's own GitLab takes the
VMs' write limits, or 16 MB/s and 160 IOPS where they are none, and the
stack is always configured with that default. The snapshot's stamp
covers the compose override and gitlab.rb too, and carries the day it
was made: the upstream's token step makes the root token for a year,
so a snapshot 300 days old is made anew. A stamp that cannot be read
fails the unit instead of removing a good snapshot. Preparation waits
within the unit's time. A folder that is no clone is refused, never
removed; the clone goes through a folder of its own, removed if it
fails, as is the token's temporary file. The token step's output is
masked before it reaches an error.
… and keep a snapshot Docker could not read

GitLab is made under the limits of the unit that makes it, as it later
runs under them. A snapshot whose stamp Docker could not read fails the
unit instead of being made anew; only a missing stamp counts as none.
By hand, prepare refuses a folder that is not the stack Nullius cloned,
and removes nothing. Website host suffix says that GitLab is served with the mock websites.
…er could not read

A snapshot is none only when Docker says there is no such volume, or
when it holds no stamp; any other failure to look for it or read it
fails the unit and keeps it. With GitLab's image gone, after a prune or
a new pin, the snapshot counts as none, so the images are pulled and
GitLab is made anew. The token step runs under timeout, so a driver
killed during it leaves no shell retrying for longer than an hour. Task
041 counts among those that need the websites option.
…y where Docker says so

The volume, GitLab's image and the snapshot's stamp count as absent
only where Docker or cat says there is no such volume, image or file,
in any case; every other failure fails the unit and keeps the snapshot.
The token step's timeout runs in the foreground and ends before
Python's, so the shell under it never outlives it.
…ls, as its other refusals do, and say that its GitLab tasks run one at a time
…s write-limit refusals, not their environment
…th each agent's own budget, show its VM provider on Where and ask its website host suffix on Dataset, for the datasets that need it
@sirotaoleg
sirotaoleg marked this pull request as ready for review November 19, 2026 02:28
aitianshi pushed a commit that referenced this pull request Nov 19, 2026
…viders, set its own limits and prove a merged null control (#133)

## What

A benchmark of the standard can now say more about how its units run and
end, which a computer-use benchmark like OSWorld2 (#134) needs. `gated`,
the new limits and `prove`'s new options are opt-in. The relay's rules
and the rules of how a unit ends change for every benchmark of the
standard; Risk lists them.

- **Gated tasks** (`gated: true`): a unit's failure that the relay's
records explain, a model endpoint's (the agent's, the judge's or the
simulated user's), says in its `result.json` the status and class alone,
never the provider's words, which can quote the request. The trial's
call records keep the provider's words, and an error a driver raises
itself is kept as the driver words it.
- **A benchmark option's values by name** (`names`: {value: name}): the
builder's cards and select, Review and the benchmark's page show the
name, `about` stays its tooltip, and a recipe and the CLI keep the
value. OSWorld1's agents are named as their upstream spells them.
- **An option on its own step** (`step`): `where`, where the work runs,
on the Where step, one value as fixed, and in Review's Where row;
`dataset`, what some datasets need, on the Dataset step for those
datasets. OSWorld1's VM provider is on Where.
- **Limits of a benchmark's own:**
- `limits.score_seconds`: how long `score` may take, the benchmark's
level below `--timeout score=` (#191); the Limits step shows it as the
benchmark's;
- `limits.call_seconds`: how long the relay waits for one model call,
also `relay serve --timeout`;
- **How a unit ends:**
- a driver's error that begins `EnvironmentSetupError: `,
`VerifierError: `, `CancelledError: ` or `CauseUnknownError: ` keeps
that kind;
- the agent's own last call refused as longer than the model's context
window is `ContextWindowExceededError`, the agent's;
- a last 429 is read from the agent's own calls in the order they began,
so a side model's later 200 does not cover it;
- a unit a guard's stop ended keeps the score its driver gave, its own
last call's 429 notwithstanding, unless that 429 said the credit is
spent. `score` can say the stop's reason (`verifier_result.stopped`);
- with #154's `--retries`, a unit a guard asked to stop and a full
context window are not tried again;
- for an upstream's own answerer: `OutputLengthExceededError` only when
its last reply has neither text nor a tool call. When the answerer fails
and its last reply was a Responses object answered 200 and `failed`,
that is an API error.
- **The relay:**
- a non-streamed 200 to a model call with no reply in it reaches the
caller and the record as a 502, or the 429 or 402 of an error it held
(the record keeps `answered: 200`). #124 leaves counting an error inside
a 200 to a pull request of its own; this one takes only a 200 with no
reply in it;
- a stream cut off midway, after the caller had its 200, is recorded as
a 502, and its booking against the spending cap stays whole;
- a call refused with 400, 413 or 422 records each inline image it sent
by its declared type, the type its bytes show and its size, never a
byte;
- a `stop` file beside the relay's records: the relay notes in it an
agent's call refused for the model's context window, and that note
refuses nothing. Once a benchmark's guard writes a line of its own
there, the agent's later calls are refused unsent, while side models and
the judge's and simulated user's routes go on;
- a failed Responses object with no usage records a cost of 0, and its
booking is kept.
- **A job's name:**
- a run holds its job name until its process ends, and one under a name
another run still holds is refused before it writes anything;
- a rerun into the same job moves each rerun trial's earlier records
(`.work/relay`, `result.json`, `verifier/`, `agent/`) aside to
`.work/earlier.<ns>`.
- **prove and merge:**
- `nullius standard prove ... --bench-option NAME=VALUE` sets the
benchmark's own options for the units check and the controls;
- `--job nop=MERGED` (or `oracle=`) checks a merged launch of a control
against its bound instead of running it, and every trial's own record;
- `merge` accepts shards that differ in benchmark options and keeps each
shard's in `merged_from`;
- `merge` refuses, unless `--force`, a trial whose record names another
job, benchmark, version, control or agent, and a job itself merged
despite differences. It lists the shards absent from a merge only when
the launch has at most 1,000.
- **Smaller:**
  - a trace that cannot be built leaves `agent/trace.error.txt`;
  - an Original's trajectory keeps its tool calls;
  - each call's record keeps the request's sha256;
  - `thinking_tokens` count as reasoning;
- a setup's `comparable` can list the models whose runs compare with the
paper, and the builder's message for another model says so;
- a test checks that every file a host's snapshot takes of a bundled
entry is package data.

**Read first:** `failure`, then `_hollow` and the stop in
`Relay._forward` (`app/nullius/standard/relay.py`). Next, the outcome
rules in `Run` (`app/nullius/standard/run.py`), then `names` and `step`
in `recipe.compile` and `benchOptions` in the builder, the `--job` check
in `prove.py` and the trial check in `merge.py`. Then
`test_standard_gated.py`, `test_standard_context_window.py` and
`test_standard_controls.py`. The rest is the definitions' checks, the
CLI options, the sidecar's argument and the pages.

## Touches

the standard (`app/nullius/standard/`: `relay`, `run`, `prove`,
`definitions`, `merge`, `job`, `cli`, `container`, `recipe`, `fidelity`,
`retries`) · the builder (`app/static/builder/`, `app/web/builder.py`:
the score's default on the Limits step, an option's names, and an option
on its own step) · the benchmark page (an option's names) · OSWorld1's
definition (its agents' names, its VM provider on Where) · docs and site
(`docs/standard.md`, `docs/cli/run.md`, `docs/cli/read-jobs.md`,
`docs/builder.md`, `docs/settings/model-providers.md`)

## Verified

- At `e768c632`, on main at `db6e8f96`, on Python 3.12: backend suite
(`python3 app/tests/parallel.py`), 2310 tests in 129 modules, OK; route
suite with Node on PATH, 639 tests in 53 modules, OK; the wheel's check
(`app/packaging/verify.py`), OK.
- **An option's names and its step, in a headless Chromium, light and
dark, 1400 and 390 px,** with #134 on top: OSWorld1's agents by name in
their select, OSWorld2's in their cards; the VM provider of each as a
fixed value on the Where step, shown only while a participant that takes
it is chosen; OSWorld2's Website host suffix on the Dataset step for All
tasks, not for Offline, no services; Served by empty on any choice of
model. No page error.
- **The 12 changed test modules on main's source** (`5bcfd82d`): 9 fail,
2 pass, and 1 does not end.
- `test_standard_context_window`: 19 of 20; `test_standard_controls`: 17
of 43; `test_standard_relay`: 8 of 58; `test_standard_gated`: 4 of 6.
- `test_standard_run`: 2 of 50; `test_standard_trace`: 2 of 20;
`test_standard_parts`, `test_standard_served_by` and
`test_standard_units`: 1 each.
- `test_standard_shard` does not end within 4 minutes: one of its merges
names 2^64 shards, and main's `merge` lists every absent one.
  - `test_standard_versions` and `test_dependency_surface` pass on both.
- **Covered in #134, not here:** that `limits.call_seconds` reaches the
relay and the sidecar's `--timeout`. OSWorld2 is the first entry to set
it (`test_osworld2_muse.py`).
- **#124's own tests.** Three were changed for two deliberate
differences: a 200 with no reply is a 502, and a failed Responses object
is read as failed. The rest of `test_standard_served_by` passes as main
has it; `write_shard` in `test_standard_shard` takes #137's `reward`
beside this branch's `bench_options`.
- `PYTHONPATH=app python3 -m nullius standard check`: ok. `python3
app/tests/catalogue_docs.py --write`, then the check: up to date.
`python3 app/tests/field_contract.py`: ok. `app/site/readme_media.py
--check`: current. The docs check reports the same two findings in
`AGENTS.md` as on main.
- Not run: a benchmark already on main, live, on this branch; the
container backend with `--timeout`; the docker tier; the site build.

## Risk

- **For every benchmark of the standard:**
- a non-streamed 200 with no reply is a 502, and a stream cut off midway
is recorded as one;
- the agent's own last call refused for its context window is the
agent's error (`ContextWindowExceededError`), where it was an API error
outside its rate;
- an agent's last 429 followed by a side model's 200 makes the unit
`rate_limited`, where its score stood;
- an upstream answerer whose last reply was a `failed` Responses object
is an API error, where it was the agent's;
- a driver's error of a kind it names: a setup error or a stop is left
out of the agent's rate, and a verifier error or one of no known cause
counts as unknown. A wrong kind would move a unit between them;
- `prove` runs no control when the units check failed, and fails a
control job with a unit that has neither a score nor an error;
- a run under a job name another run still holds is refused, and a rerun
into an existing job moves the rerun trials' records aside;
- `merge` refuses a trial whose record names another job unless
`--force`.
- **Records.** Stored records gain fields and keep the old ones.
- These commits sit on main at `db6e8f96`, with #124, #150, #191, #154,
#174 and the Recipes page (#205). #124's `--served-by` code is main's as
merged; #191's time limits come first, so `--timeout score=` is used
over `limits.score_seconds`, and a unit's cap on model calls
(`UnitCallCapError`) is read before this PR's outcome rules. #154's
`--retries` replaces the one more run of a rate-limited unit, so the
earlier `limits.run_again: false` that switched it off is gone; main
counts a provider's 402 as a usage limit, and the tests follow. The
first and fourth commits gave a definition pins of its serving
providers, defaults of Served by after review; the last commit removes
them, as the reviewer asked, since no benchmark declares one: the diff
over main has none, and Served by is the user's alone, as in #124.
- The fifth commit shortens this PR's docs and messages. No logic
changes.
- **Names and Where.** An option's names change only what is shown; a
recipe saved before keeps working. An option of where the work runs
leaves the Agent step for Where, and one some datasets need is asked on
Dataset: OSWorld1's VM provider is now on Where.
- Undone by reverting the commits.

---------

Co-authored-by: Oleg <247799477+sirotaoleg@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants