Skip to content

feat(standard): run each unit's agent in a Boat VM of its own - #150

Merged
aitianshi merged 9 commits into
mainfrom
research/boat-sandbox-fit
Nov 17, 2026
Merged

aitianshi merged 9 commits into
mainfrom
research/boat-sandbox-fit

Conversation

@aitianshi

@aitianshi aitianshi commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #149

What

Adds Boat as a third backend for runs of the standard. Each unit's agent or capability runs in a boat.dev VM of its own, started where the run runs (this machine or a Settings host) with the key saved here in Settings → Sandbox.

How a unit runs on Boat

  • The agent runs as an unprivileged user in the VM. iptables limits it to loopback, with DNS refused, and the unit's relay, beside the run, is reached through an ssh -R tunnel, so the VM never holds a model key.
  • The agent's environment, which holds the relay's word and any licence value, goes to the VM as a file only the agent's user reads, and its shell removes it. It is never on a command line, here or in the VM.
  • The agent's install, its definition and its runtimes go to the VM before the agent's time starts. Its home and workspace are sent as it starts, after its setup writes its config. Results come back with rsync --safe-links.
  • A check that executes the agent's code (score.runs_code: HumanEval, the workspace kit) runs it back where the run runs, in the throwaway check container, never as the user. Review asks for Docker there, and the run is refused without it.
  • The VM's auto-stop follows the unit's time limit. A free trial caps it at 2 hours, and the run says so once when a unit could outlast that; later units then ask for the 2 hours at once.
  • The VM is always deleted, whatever ends the unit. On a Stop, nothing is fetched back, and a Stop also ends the wait for a VM to come up. A delete Boat refuses is tried once more, then said and kept as delete_error in agent/boat.json.

Refusals and warnings

  • A runtime link that would send the user's home, keys, a tool's login folder (~/.config, ~/.local, ~/.docker, …) or Nullius's storage to a VM is refused.
  • A unit whose setup gives the agent MCP servers, or needs its own image, ends as not run, with the reason.
  • The run uses no more units at a time than the account can run VMs at once.
  • Before starting, the run says when the account can start fewer VMs this hour or today than it has units. A unit waits up to an hour for a free start, and a Stop ends the wait.

The key

  • The Boat key is withheld from every command Nullius runs, Harbor runs included. Only a standard run on Boat receives it, over stdin like the agent keys, on this machine or a host. A host is checked for ssh, ssh-keygen, rsync and tar among its prerequisites.

Interface

  • Settings → Sandbox adds the Boat key, edits it and removes it; Remove asks first.
  • Builder, Where step: Boat is offered on this machine and on hosts, with the VM sizes the account can start; the others are greyed out. A backend that cannot be used says so under its name, so the name stays whole.
  • Review: the Where line names the size, and Review refuses a size the account cannot start.
  • Logo: Boat's mark switches to its white version on a dark background.

Touches

backend · routes · templates and static · docs and site · the standard

  • app/nullius/standard/boat.py (new), run.py, recipe.py, job.py, cli.py, kits/agent-workspace-v1/standard.json (runs_code)
  • app/nullius/execution.py (the Boat key withheld, handed to a Boat run), prerequisites.py (a host's Boat programs)
  • app/web/execute.py, builder.py, settings.py
  • app/static/builder/, app/static/settings/sandbox.js, app/templates/settings/settings.html, app/static/style.css, app/static/brand/boat.svg and boat-dark.svg (Boat's own marks, copied unchanged)
  • app/tests/test_standard_boat.py, app/tests/test_execution.py, app/tests/test_prerequisites.py, app/ui_tests/test_settings_page.py, app/ui_tests/test_agents.py
  • docs/standard.md, docs/builder.md, docs/cli/run.md, docs/settings/sandbox.md; AGENTS.md (a Documentation convention: docs/ stays terse)

Before and after

Screenshots were taken locally, in light at 1280 px and dark at 390 px:

  • Settings → Sandbox: before, no Boat card. After, a Boat card with Add key, Edit, and Remove with its confirmation.
  • Builder, Where step: before, Local processes and Container. After, a Boat tile and a VM size row: Small and Default can be chosen, Large and XLarge read "Not on this account" on a trial.
  • Review: after, the Where line names the size ("small VMs"), and a size the account cannot start is refused: "Boat cannot start a large VM on this account: … Choose another size."

Verified

Tests, on 5cd922b (main) plus this branch:

  • Backend suite: every module passes, each run alone with a 5-minute limit (the parallel runner hangs on this machine); 2160 tests before the host change.
  • Route suite: 530 tests pass.
  • test_standard_boat.py: 36 tests with a stand-in VM. Besides the run itself they cover:
    • the firewall's rule order;
    • the environment sent on stdin, never on argv;
    • a Stop while a VM comes up;
    • refused runtime links, ~/.local and ~/.config included;
    • a delete Boat refuses;
    • the trial's TTL, remembered.
  • test_execution.py: only a standard run on Boat is handed the Boat key.

Real runs on a free-trial Boat account

  • HumanEval, 2 problems, model meta/muse-spark-1.3-contributor, --max-usd 0.10.
    • Before the review fixes, every agent scored 2/2:

      Agent VM and install Agent time
      opencode 55 s 10 to 12 s
      codex 76 to 80 s 13 to 17 s
      goose 37 to 55 s 17 to 21 s
      reference 6 to 9 s 11 to 17 s
    • After the fixes, and again after the rebase onto main: opencode scored 2/2 three times. In the two runs whose trials were read, each recorded check: container.

  • In one real VM, as the agent's user:
    • name lookup failed (dns=2);
    • the internet failed (internet=7);
    • loopback answered (loopback=0);
    • iptables -S OUTPUT listed the DNS rejects before the loopback accept.
  • No VM was left after any run (GET /sandboxes returned []). Model spend was about $0.02 in all.
  • The starts warning against the real account:
    • 20 units: "Boat can start 9 more VMs this hour on this account: the other units wait for the next hour"
    • 164 units: "… 105 of the 164 units will end without running"

Browser

  • Settings, the Where step and Review, in light at 1280 px and dark at 390 px: no page errors.
  • The Boat mark shows its white version in dark and the blue one in light.

Not verified

  • No real Boat run from a host: the key's hand-over to a host and the host's prerequisites are tested with a stand-in host.
  • No paid-plan account: Large and XLarge sizes, and auto-stops over 2 hours, are untested.
  • A delete Boat refuses was only seen with a stand-in; no real delete failed.

Risk

  • The local and container backends are untouched except for shared plumbing in run.py: the unit's VM context and the agent-time ordering.
  • The workspace kit's check now declares runs_code. That changes nothing on the local or container backends; it only makes a Boat run of the kit need Docker.
  • If Boat's API changes, Boat runs end with EnvironmentSetupError and the other backends are unaffected.
  • A VM that outlived a crash of this machine is stopped by Boat's own auto-stop.
  • Undo by reverting the commits; nothing is migrated.

@aitianshi aitianshi self-assigned this Nov 16, 2026
@github-actions

This comment has been minimized.

Adds the boat backend (app/nullius/standard/boat.py): one boat.dev VM per
unit, the agent as a user locked to the VM's loopback, its model calls
through a reverse tunnel to the unit's relay on this machine, the workspace
copied back without links that leave it, the VM deleted after. Concurrency
is capped to the VMs the account runs at once. The builder's Where step
offers Boat; Settings > Sandbox keeps its API key with Edit and Remove.
…ot start every unit

Send the install before the agent's time and the home and workspace as it
starts; follow the unit's time with the VM's auto-stop, within a trial's
2 hours; refuse links that would send this user's own files; delete a VM
whatever ends the unit, and skip the fetch on a Stop. Review refuses a size
the account cannot start, Remove asks first, the run says when the account
can start fewer VMs this hour or today than it has units, and the Boat mark
has its own version on a dark background.
- A Boat run whose check executes the agent's code checks it in the
  throwaway check container, and is refused without Docker; it ran as the
  user on this machine. The workspace kit's check declares runs_code.
- A unit whose setup gives the agent MCP servers is not run on Boat: they
  start from this machine's paths, which the VM does not have.
- The agent's user's firewall refuses DNS before it accepts loopback; the
  rules were inserted in an order that let DNS through.
- The agent's environment (the relay's word, a licence's value) goes to the
  VM as a file its user reads and removes, never on an ssh command line.
- A Stop ends the wait for a VM to come up, and the VM is deleted.
- Runtimes linked from ~/.local, ~/.cache, ~/.config and other tools'
  login folders are refused, with the Backend's own environment.
- The Boat key is withheld from every run's environment and handed over
  stdin only to a standard run on Boat.
- Once a trial's 2-hour cut is known, later units ask for it at once.
A delete is tried twice, and a VM already gone counts as deleted. One
Boat still refuses is said, with when it stops by itself, and kept as
delete_error in the unit's agent/boat.json; the end-of-run sweep says
each VM of the run it could not delete.
@aitianshi
aitianshi force-pushed the research/boat-sandbox-fit branch from f1368d9 to bf70b27 Compare November 17, 2026 15:39
@github-actions

This comment has been minimized.

The section's table was the mechanism, cell by cell. It now keeps the
guarantees, the requirements and the refusals; the builder, CLI and
Settings rows say only what Boat is and where it is set.
@github-actions

This comment has been minimized.

Why a backend cannot be used sat beside its name without wrapping, which
squeezed the name until it broke mid-word (Containe/r) once Boat made the
row three tiles. It now goes under the name; a row's tiles stay one height.
@github-actions

This comment has been minimized.

A run on a host starts its Boat VMs from there. The key stays in this
machine's store and reaches the run over stdin, as the agent keys do;
Review checks it here, and the host's ssh, ssh-keygen, rsync and tar
among its prerequisites. A Boat run whose check runs the agent's code
now asks Review for Docker where it runs.
@github-actions

This comment has been minimized.

The Boat section drops what is mechanism (how the key travels, what a
host is checked for) and keeps what a user decides on. AGENTS.md gains a
Documentation convention so later changes keep the same standard.
@github-actions

This comment has been minimized.

A run started by Nullius keeps only its provider's key and endpoint of
the environment its launcher gives it (run_env), which dropped the Boat
key handed over stdin. Here the run fell back to the key saved in
Settings; a host has none, so a run on Boat there was refused for want
of a key.
@github-actions

Copy link
Copy Markdown

🔴 PR judge · 5 of 8 yes · #150 ↔ #149

Merge risk 🔴 MAJOR the judge's reading of the diff; app/nullius/execution.py is in the path of every run
Priority 🔴 EXTREME the deepest path is app/nullius/execution.py
Question Verdict
Changes do what the issue asks · 11 of 11 criteria ✅ yes
Issue and pull request describe the same change ✅ yes
Description matches the diff, no more, no less ⚠️ partly
Diff stays in scope ✅ yes
Tests cover the change, or Verified says how ✅ yes
Docs follow the change ✅ yes
Risk named in the description ⚠️ partly
Before and after shown · 0 images · the diff touches what is visible ❌ no
Acceptance criteria, 11 of 11 met
# Criterion Verdict
1 nullius standard run ... --backend boat [--boat-size SIZE] runs each unit's agent or capability in its own Boat VM, a… ✅ met (cut)
2 The agent in the VM reaches only its relay; no model key is sent to the VM ✅ met (cut)
3 Every VM a run starts is deleted, on success, failure, timeout and Stop ✅ met (cut)
4 The VM's auto-stop follows the unit's time limit, within the 2 hours a free trial allows ✅ met (cut)
5 Time spent starting the VM and installing the agent is not counted as the agent's time ✅ met (cut)
6 Nothing under the user's own private folders (home, keys, Nullius config) is sent to a VM through a link ✅ met (cut)
7 Settings → Sandbox saves, edits and removes the Boat key; Remove asks first ✅ met (cut)
8 The builder's Where step offers Boat only on this machine, with the account's VM sizes; Review refuses a size the accou… ✅ met (cut)
9 A run says when the account can start fewer VMs this hour or today than the run has units ✅ met (cut)
10 Units that need a benchmark's MCP server or their own image end as not run on Boat, with the reason ✅ met (cut)
11 docs/standard.md, docs/builder.md and docs/cli/run.md describe the backend ✅ met (cut)
Checks the script runs itself, 4 of 4 pass
No machine attribution trailer in the commits or the description ✓
New code under app/, docs/, .github/ or .claude/ ✓
No Flask import under the backend package ✓
No key-shaped string in the diff ✓

some of the diff was cut before the judge read it

@aitianshi
aitianshi merged commit 2815e65 into main Nov 17, 2026
2 checks passed
@aitianshi
aitianshi deleted the research/boat-sandbox-fit branch November 17, 2026 16:10
aitianshi added a commit that referenced this pull request Nov 17, 2026
… work on a unit and cap its model calls (#191)

Closes #190

## What

A standard run can now force its limits, and the agent's time limit
covers its whole work on a unit.

- **Time limits by phase.** `run --timeout PHASE=SECONDS` forces the
limit of a phase: `setup`, `image`, `agent`, `verify` or `score`. Empty
keeps the default, which comes from the task, then the benchmark, then
Nullius. `config.json` records what was forced under
`standard.timeouts`.
- **One agent limit per unit.** The agent's limit is one deadline for
its whole work on a unit: every turn of a conversation, and the
simulated user's time between them. Before, it applied to each reply. A
capability keeps one limit per add and per recall, because a memory adds
a unit's sessions one by one. τ³-bench's `limits.turn_seconds` goes from
600 s a reply to 7200 s a conversation.
- **Turns.** `run --max-turns N` sets the turns of a conversation. The
builder offers it only on a benchmark that holds conversations.
- **A cap on model calls.** `run --max-calls N` caps the model calls an
agent makes on a unit. The relay counts each answered call on the
agent's route; rate-limited and failed calls are not counted, and
neither are the judge's or the simulator's. Past the cap the relay
refuses the call with a 402, and the unit ends as `UnitCallCapError`, an
agent error that is not scored. It is for agents only: a capability's
route also carries the benchmark's reader, so a capability run refuses
it.
- **A benchmark's own bounds.** They are now options marked `limit`.
τ³-bench gets Steps per conversation (`--max-steps`, from 1 up to its
default of 200) and Tool errors per conversation (`--max-errors`).
OSWorld gets Steps per task (each agent script's `--max_steps`); left
empty it uses the chosen agent's own value, declared with the new `by`
key.
- **Builder, Limits step.** It shows Scope, then Time limits in minutes,
then a section named after the benchmark when it has bounds of its own.
An empty field shows the value the run uses: in the text colour when the
benchmark or the agent declares it, grey when it is Nullius's default
(priority: Nullius < agent < benchmark). Nothing is typed into a field
for the user.

## Touches

- `app/nullius/standard/`: `run.py`, `relay.py`, `container.py`,
`job.py`, `cli.py`, `recipe.py`, `definitions.py`
- `app/nullius/runs/classify.py`: `UnitCallCapError` is an agent error
- Catalogue: τ³-bench and OSWorld's `standard.json`, their drivers and
`upstream-settings.json`
- `app/static/builder/`: the Limits step
- Docs: `docs/cli/run.md`, `docs/standard.md`, `docs/builder.md`, the
field guide and the generated supported-benchmark pages

## Before and after

The Limits step for τ³-bench, before and after:

| Before | After |
| --- | --- |
| <img width="440" alt="Limits step on main: Scope only, with tasks,
concurrency, attempts and two spending caps"
src="https://github.com/user-attachments/assets/80b468a4-326f-4057-a4b2-5551ea93e642"
/> | <img width="440" alt="Limits step on this branch: Scope with turns
and model calls, Time limits in minutes, then a tau3-bench section with
steps and tool errors"
src="https://github.com/user-attachments/assets/6278494c-6c9f-456f-b739-9578494315ee"
/> |

## Verified

- Backend suite: 2175 tests OK. Route suite: 531 tests OK (after merging
`main`, with #124's serving providers and #150's Boat VMs).
- New tests:
- A whole conversation that ends past the agent limit, though no single
turn does.
  - Four one-call turns cut on the fourth call by `max_calls=3`.
  - The relay counts only answered calls on the agent's route.
- A slow capability whose three adds each fit a 1 s limit while their
total does not.
  - A capability run refuses `--max-calls`.
- Recipe validation of the limits, including a value that rounds to 0 s.
  - τ³-bench steps above 200 are refused.
- OSWorld's `--max_steps` is replaced, or added when the agent's command
doesn't set it.
- Against the installed τ³-bench upstream: the driver uses 200 steps and
10 errors by default, and 7 and 3 when the run sets them.
- Checked in the browser, light and dark, for τ³-bench and OSWorld.

## Risk

- **Agent timeouts on conversation benchmarks.** A conversation that
used to fit 600 s per reply now has a single limit for the whole
conversation, 7200 s on τ³-bench. A very long conversation can now time
out where it didn't before. Undo by forcing a larger `--timeout
agent=SECONDS`, or by reverting `limits.turn_seconds`.
- **Comparability.** A run whose limits differ from the defaults is not
yet marked as not comparable with the leaderboard.
- **Overlap.** #174 (the agent's setup timed apart from its run) and
#133 (a benchmark's own limits) change the same parts of `run.py` and
the catalogue, so expect merge conflicts.
aitianshi pushed a commit that referenced this pull request Nov 19, 2026
…viders, set its own limits and prove a merged null control (#133)

## What

A benchmark of the standard can now say more about how its units run and
end, which a computer-use benchmark like OSWorld2 (#134) needs. `gated`,
the new limits and `prove`'s new options are opt-in. The relay's rules
and the rules of how a unit ends change for every benchmark of the
standard; Risk lists them.

- **Gated tasks** (`gated: true`): a unit's failure that the relay's
records explain, a model endpoint's (the agent's, the judge's or the
simulated user's), says in its `result.json` the status and class alone,
never the provider's words, which can quote the request. The trial's
call records keep the provider's words, and an error a driver raises
itself is kept as the driver words it.
- **A benchmark option's values by name** (`names`: {value: name}): the
builder's cards and select, Review and the benchmark's page show the
name, `about` stays its tooltip, and a recipe and the CLI keep the
value. OSWorld1's agents are named as their upstream spells them.
- **An option on its own step** (`step`): `where`, where the work runs,
on the Where step, one value as fixed, and in Review's Where row;
`dataset`, what some datasets need, on the Dataset step for those
datasets. OSWorld1's VM provider is on Where.
- **Limits of a benchmark's own:**
- `limits.score_seconds`: how long `score` may take, the benchmark's
level below `--timeout score=` (#191); the Limits step shows it as the
benchmark's;
- `limits.call_seconds`: how long the relay waits for one model call,
also `relay serve --timeout`;
- **How a unit ends:**
- a driver's error that begins `EnvironmentSetupError: `,
`VerifierError: `, `CancelledError: ` or `CauseUnknownError: ` keeps
that kind;
- the agent's own last call refused as longer than the model's context
window is `ContextWindowExceededError`, the agent's;
- a last 429 is read from the agent's own calls in the order they began,
so a side model's later 200 does not cover it;
- a unit a guard's stop ended keeps the score its driver gave, its own
last call's 429 notwithstanding, unless that 429 said the credit is
spent. `score` can say the stop's reason (`verifier_result.stopped`);
- with #154's `--retries`, a unit a guard asked to stop and a full
context window are not tried again;
- for an upstream's own answerer: `OutputLengthExceededError` only when
its last reply has neither text nor a tool call. When the answerer fails
and its last reply was a Responses object answered 200 and `failed`,
that is an API error.
- **The relay:**
- a non-streamed 200 to a model call with no reply in it reaches the
caller and the record as a 502, or the 429 or 402 of an error it held
(the record keeps `answered: 200`). #124 leaves counting an error inside
a 200 to a pull request of its own; this one takes only a 200 with no
reply in it;
- a stream cut off midway, after the caller had its 200, is recorded as
a 502, and its booking against the spending cap stays whole;
- a call refused with 400, 413 or 422 records each inline image it sent
by its declared type, the type its bytes show and its size, never a
byte;
- a `stop` file beside the relay's records: the relay notes in it an
agent's call refused for the model's context window, and that note
refuses nothing. Once a benchmark's guard writes a line of its own
there, the agent's later calls are refused unsent, while side models and
the judge's and simulated user's routes go on;
- a failed Responses object with no usage records a cost of 0, and its
booking is kept.
- **A job's name:**
- a run holds its job name until its process ends, and one under a name
another run still holds is refused before it writes anything;
- a rerun into the same job moves each rerun trial's earlier records
(`.work/relay`, `result.json`, `verifier/`, `agent/`) aside to
`.work/earlier.<ns>`.
- **prove and merge:**
- `nullius standard prove ... --bench-option NAME=VALUE` sets the
benchmark's own options for the units check and the controls;
- `--job nop=MERGED` (or `oracle=`) checks a merged launch of a control
against its bound instead of running it, and every trial's own record;
- `merge` accepts shards that differ in benchmark options and keeps each
shard's in `merged_from`;
- `merge` refuses, unless `--force`, a trial whose record names another
job, benchmark, version, control or agent, and a job itself merged
despite differences. It lists the shards absent from a merge only when
the launch has at most 1,000.
- **Smaller:**
  - a trace that cannot be built leaves `agent/trace.error.txt`;
  - an Original's trajectory keeps its tool calls;
  - each call's record keeps the request's sha256;
  - `thinking_tokens` count as reasoning;
- a setup's `comparable` can list the models whose runs compare with the
paper, and the builder's message for another model says so;
- a test checks that every file a host's snapshot takes of a bundled
entry is package data.

**Read first:** `failure`, then `_hollow` and the stop in
`Relay._forward` (`app/nullius/standard/relay.py`). Next, the outcome
rules in `Run` (`app/nullius/standard/run.py`), then `names` and `step`
in `recipe.compile` and `benchOptions` in the builder, the `--job` check
in `prove.py` and the trial check in `merge.py`. Then
`test_standard_gated.py`, `test_standard_context_window.py` and
`test_standard_controls.py`. The rest is the definitions' checks, the
CLI options, the sidecar's argument and the pages.

## Touches

the standard (`app/nullius/standard/`: `relay`, `run`, `prove`,
`definitions`, `merge`, `job`, `cli`, `container`, `recipe`, `fidelity`,
`retries`) · the builder (`app/static/builder/`, `app/web/builder.py`:
the score's default on the Limits step, an option's names, and an option
on its own step) · the benchmark page (an option's names) · OSWorld1's
definition (its agents' names, its VM provider on Where) · docs and site
(`docs/standard.md`, `docs/cli/run.md`, `docs/cli/read-jobs.md`,
`docs/builder.md`, `docs/settings/model-providers.md`)

## Verified

- At `e768c632`, on main at `db6e8f96`, on Python 3.12: backend suite
(`python3 app/tests/parallel.py`), 2310 tests in 129 modules, OK; route
suite with Node on PATH, 639 tests in 53 modules, OK; the wheel's check
(`app/packaging/verify.py`), OK.
- **An option's names and its step, in a headless Chromium, light and
dark, 1400 and 390 px,** with #134 on top: OSWorld1's agents by name in
their select, OSWorld2's in their cards; the VM provider of each as a
fixed value on the Where step, shown only while a participant that takes
it is chosen; OSWorld2's Website host suffix on the Dataset step for All
tasks, not for Offline, no services; Served by empty on any choice of
model. No page error.
- **The 12 changed test modules on main's source** (`5bcfd82d`): 9 fail,
2 pass, and 1 does not end.
- `test_standard_context_window`: 19 of 20; `test_standard_controls`: 17
of 43; `test_standard_relay`: 8 of 58; `test_standard_gated`: 4 of 6.
- `test_standard_run`: 2 of 50; `test_standard_trace`: 2 of 20;
`test_standard_parts`, `test_standard_served_by` and
`test_standard_units`: 1 each.
- `test_standard_shard` does not end within 4 minutes: one of its merges
names 2^64 shards, and main's `merge` lists every absent one.
  - `test_standard_versions` and `test_dependency_surface` pass on both.
- **Covered in #134, not here:** that `limits.call_seconds` reaches the
relay and the sidecar's `--timeout`. OSWorld2 is the first entry to set
it (`test_osworld2_muse.py`).
- **#124's own tests.** Three were changed for two deliberate
differences: a 200 with no reply is a 502, and a failed Responses object
is read as failed. The rest of `test_standard_served_by` passes as main
has it; `write_shard` in `test_standard_shard` takes #137's `reward`
beside this branch's `bench_options`.
- `PYTHONPATH=app python3 -m nullius standard check`: ok. `python3
app/tests/catalogue_docs.py --write`, then the check: up to date.
`python3 app/tests/field_contract.py`: ok. `app/site/readme_media.py
--check`: current. The docs check reports the same two findings in
`AGENTS.md` as on main.
- Not run: a benchmark already on main, live, on this branch; the
container backend with `--timeout`; the docker tier; the site build.

## Risk

- **For every benchmark of the standard:**
- a non-streamed 200 with no reply is a 502, and a stream cut off midway
is recorded as one;
- the agent's own last call refused for its context window is the
agent's error (`ContextWindowExceededError`), where it was an API error
outside its rate;
- an agent's last 429 followed by a side model's 200 makes the unit
`rate_limited`, where its score stood;
- an upstream answerer whose last reply was a `failed` Responses object
is an API error, where it was the agent's;
- a driver's error of a kind it names: a setup error or a stop is left
out of the agent's rate, and a verifier error or one of no known cause
counts as unknown. A wrong kind would move a unit between them;
- `prove` runs no control when the units check failed, and fails a
control job with a unit that has neither a score nor an error;
- a run under a job name another run still holds is refused, and a rerun
into an existing job moves the rerun trials' records aside;
- `merge` refuses a trial whose record names another job unless
`--force`.
- **Records.** Stored records gain fields and keep the old ones.
- These commits sit on main at `db6e8f96`, with #124, #150, #191, #154,
#174 and the Recipes page (#205). #124's `--served-by` code is main's as
merged; #191's time limits come first, so `--timeout score=` is used
over `limits.score_seconds`, and a unit's cap on model calls
(`UnitCallCapError`) is read before this PR's outcome rules. #154's
`--retries` replaces the one more run of a rate-limited unit, so the
earlier `limits.run_again: false` that switched it off is gone; main
counts a provider's 402 as a usage limit, and the tests follow. The
first and fourth commits gave a definition pins of its serving
providers, defaults of Served by after review; the last commit removes
them, as the reviewer asked, since no benchmark declares one: the diff
over main has none, and Served by is the user's alone, as in #124.
- The fifth commit shortens this PR's docs and messages. No logic
changes.
- **Names and Where.** An option's names change only what is shown; a
recipe saved before keeps working. An option of where the work runs
leaves the Agent step for Where, and one some datasets need is asked on
Dataset: OSWorld1's VM provider is now on Where.
- Undone by reverting the commits.

---------

Co-authored-by: Oleg <247799477+sirotaoleg@users.noreply.github.com>
aitianshi added a commit that referenced this pull request Nov 19, 2026
…ces on the machine and a null control (#134)

Stacked on #133: this pull request's own commits are the last sixteen.
Until #133 is merged, the diff shows its files too. It can be merged
after #133.

## What

OSWorld-V2's 108 tasks become a bundled benchmark, `osworld2`, on the
Nullius standard, at the upstream's release `osworld-v2.1`. Each task is
a unit in an Ubuntu VM of its own on the upstream's docker provider, set
up by the task's own class and scored from 0 to 1 by its own evaluate().
The tasks and their assets are gated on Hugging Face. Nullius does not
fetch them, and the entry is `gated` (#133): a unit's failure from a
model endpoint says its status alone, and the driver names an upstream
exception by a known class's name at most, never by its words. The
Original setup runs one of OSWorld-V2's own agents, M3 or Muse Spark,
chosen by the Agent option, from its published script, with every model
call sent to the relay. Claude Sonnet 4.6 is the LLM grader of 21 tasks
and the simulated user of 6. No provider is pinned: Served by is the
user's, as for OSWorld1. Only M3's runs count as comparable with the
paper's row. Its Review row and its generated page name the setup
"OSWorld-V2 agents". A null control runs one `DONE` action, then the
task's evaluate() on the untouched VM. The entry declares
`agent.computer/v1`. No agent from the Agents page runs it yet. Its
texts are one Good to know line and 9 one-line limitations.

The mock websites of 40 tasks are OSWorld-web at the same tag. The
guide's steps build them from source and serve them on the machine the
units run on. A unit that needs them is refused before its VM starts
when they do not answer. For 026 and 041, Nullius makes GitLab on the
run's machine before the first of them, with a root token it keeps
there, then starts it anew from its snapshot for each, one task at a
time; Overleaf's 2 tasks get Overleaf started anew. Each of the 8 tasks
that ask for a proxy gets a proxy of its own. The options are the agent,
by name, Thinking for M3, and disk write limits for each VM. The website
host suffix is asked on the Dataset step, for the four datasets with
website or GitLab tasks. Steps per task is with the limits, each agent's
own by default, as its wrapper runs it: 100 for M3, 500 for Muse Spark.
The VM provider, Docker alone, is on the Where step. By default M3 takes
100 steps with thinking off, as the upstream's M3 script does; the paper
ran M3 at 500 steps with thinking on. Muse Spark runs as published at
this release, while its published score came from newer upstream code; a
limitation says it does not compare, and a docs page says what differs.
There are six datasets: All tasks (108, the upstream's `test_v2.json`),
Offline (82), and four by the services their tasks need. Tasks 103 and
104 are not supported: their evaluate() needs FreeCAD on the driver's
machine. Review and the run's start ask for Docker and for the scoring
programs (ImageMagick, ffmpeg, tesseract) on the driver's machine. As
for OSWorld1, the Where step greys out the container card under the
Original, whose docker provider needs Docker on the machine. A guide,
`docs/benchmarks/osworld2-machine.md`, prepares a machine, runs the null
control on one task as its last setup step, and says what a run takes in
time and money. A second page, `docs/benchmarks/osworld2-results.md`,
says how a run differs from the paper and the upstream, and how a unit
is scored, stopped or left unscored.

## Touches

standard (an option's example and a machine setup's label: the
definition check, the benchmark and agent pages, the builder, Review) ·
catalogue · templates and static (the mark) · docs and site
(`docs/benchmarks/`, the generated pages, `AGENTS.md`) · routes (tests
only) · packaging (the entry's `.yml`, `.Dockerfile` and `startup/*.py`
as package data in `pyproject.toml`)

## Before and after

**Before:** The Benchmarks page before this change

![Benchmarks page
before](https://github.com/user-attachments/assets/718bb2be-441a-48e3-a0d6-dccea074f984)

**After:** OSWorld2 listed, with its mark

![Benchmarks page
after](https://github.com/user-attachments/assets/60e3c777-dd18-44de-b755-36e60fe3b0f1)

**Before:** The builder's Benchmark step under Computer use: OSWorld1,
beside the selected GAIA

![Benchmark step
before](https://github.com/user-attachments/assets/dfad41fe-fb70-4662-bd2f-56ee25df010d)

**After:** OSWorld1 and OSWorld2

![Benchmark step
after](https://github.com/user-attachments/assets/e122bfcf-ff51-4028-91c4-669330738459)

**Before:** The Agents page: "n/7 benchmarks it runs"

![Agents page
before](https://github.com/user-attachments/assets/808c7fba-cd24-48d9-b52d-71ef90717a09)

**After:** "n/8": OSWorld2 counts, and no agent runs it

![Agents page
after](https://github.com/user-attachments/assets/e1ae840a-c60f-4050-8ef4-3c4ac594750c)

**Before:** An agent's page, Codex: the benchmarks it runs

![Codex's page
before](https://github.com/user-attachments/assets/5a435d9b-e15b-49d0-9bb1-63706f1e1d18)

**After:** OSWorld2 listed: it needs `agent.computer/v1`, which Codex
does not offer

![Codex's page
after](https://github.com/user-attachments/assets/a07ef2b2-82e0-4284-8ce1-05de885ca1bd)

**New:** OSWorld2's page: its setup and its proofs by dataset

![OSWorld2's
page](https://github.com/user-attachments/assets/9ee0d906-9bad-4b34-a8fc-961ab9f95a63)

**New:** Its options: the agent by name, steps, thinking, the VM
provider, the website host suffix and the disk write limits

![OSWorld2's
options](https://github.com/user-attachments/assets/8f3dbf48-e642-4312-a22d-53ebbfb3d56a)

**New:** Its dataset and its Good to know line

![OSWorld2's
dataset](https://github.com/user-attachments/assets/5d94c401-9cd3-4df9-9a25-6d75cff43ca6)

**New:** Its limitations

![OSWorld2's
limitations](https://github.com/user-attachments/assets/7559fbcc-3a75-42aa-877c-7dd8a3d793af)

**New:** Dataset step: six datasets with their controls; All tasks
chosen asks for the website host suffix, and for no token

![Dataset
step](https://github.com/user-attachments/assets/c950ee7d-4fa1-4b4a-9f19-db24ad4c57fb)

**New:** Agent step: OSWorld-V2 agents; the Agents page's agents lack
computer use; the settings every agent shares, the disk write limits

![Agent
step](https://github.com/user-attachments/assets/0a10ee0a-c924-4fac-a156-1e65c2dc25c8)

**New:** The Original's own settings: MiniMax M3 or Muse Spark by name,
and thinking

![Agent step, the Original's
settings](https://github.com/user-attachments/assets/1fe7b5d1-780c-44e4-baa7-6e58a57a2400)

**New:** Muse Spark chosen: Thinking, an M3 setting, is gone

![Agent step with Muse
Spark](https://github.com/user-attachments/assets/72910ec5-29ab-458e-b737-029f5b60f174)

**New:** Model step, Like the paper: MiniMax-M3 on OpenRouter, Claude
Sonnet 4.6 as the judge, each served by any provider until one is named

![Model
step](https://github.com/user-attachments/assets/2e3db9e9-53f3-4586-8fc0-9c481a816b47)

**New:** Limits step: Steps per task, each agent's own by default, 100
for MiniMax M3

![Limits
step](https://github.com/user-attachments/assets/cd84ad6a-de7b-4c31-8d9c-7f7d645ce7f8)

**New:** Where step: the VM provider, Docker alone; no container or Boat
backend here; Docker and the scoring programs checked on this machine

![Where
step](https://github.com/user-attachments/assets/86a5af3a-ae54-412d-9a9b-8703e75c4d06)

**New:** Review: every decision, both prerequisites ready, no model key
in this fresh profile

![Review
step](https://github.com/user-attachments/assets/459653db-56c0-4aa7-a394-cbab9d5fbd56)

**New:** The null control's merged job on test_v2, its first rows

![Null control
job](https://github.com/user-attachments/assets/7d5338e5-1a0b-40d7-86ad-008eba176607)

**New:** 052 by the test stand's model in M3: 1.0

![Job of
052](https://github.com/user-attachments/assets/097f88a5-ee73-4cbb-ac11-cb5655e749ce)

Taken at 1400 px, light theme, on a fresh profile. Cut to part of their
page: the Benchmark step and Agents pairs (the top 1000 px), Codex's
page (the top 1100 px), OSWorld2's page (four parts) and the null
control's job (its first rows). The two jobs are runs named under
Verified, opened in the app. The Model step, Review, OSWorld2's options,
dataset and limitations, and the Dataset, Agent and Where steps were
taken again after main's Served by (#124), Boat (#150) and time limits
(#191), the shorter texts and GitLab made by Nullius, and checked at
`273cca5e`: the other pictures differ there only in the sidebar, which
they leave out. The two jobs were taken again at `c4427915`, on main at
`0d83e393`, whose job page names the judge's button anew (#196); the
other pages and steps pictured are the same there. OSWorld2's options
and limitations and the Dataset, Agent, Model, Limits, Where and Review
steps were taken again at `9526a5b1`, after the agents' names, the steps
as a limit, the VM provider on Where, the website host suffix on Dataset
and Served by with no default.

## Verified

- At `9526a5b1`, on #133 (`e768c632`) on main at `db6e8f96`, on Python
3.12: backend suite (`python3 app/tests/parallel.py`), 2688 tests in 144
modules, OK; route suite with Node on PATH, 643 tests in 53 modules, OK;
the wheel's check (`app/packaging/verify.py`), OK.
- **OSWorld2's 15 test modules,** 375 tests, OK; they need the entry, so
none passes without it. 10 are skipped here: they run only when the
environment names the upstream with its gated task classes, OSWorld-web
or GitLab. Three route test modules and three backend test modules
changed for the new entry: the benchmark lists and counts, the Where
step's container card, Settings' custom keys, and OSWorld2's case of
gated tasks (`test_standard_gated.py`). Option examples and setup labels
change `test_standard_options.py`, `test_prerequisites.py` and
`test_standard_bench_options.py`, and the route modules
`test_catalogue_routes.py`, `test_agents.py` and
`test_builder_where.py`.
- At `9526a5b1`, in a headless Chromium, light and dark, 1400 and 390
px: the builder's Agent step (the agents by name, Steps no longer
there), Model step (Any of your choice: no provider filled in), Limits
step (Steps per task, 100 for M3 and 500 for Muse Spark), Where step (VM
provider docker, fixed) and Review; no page error.
- In a headless Chromium, light and dark, 1400 and 390 px: the
Benchmarks page, OSWorld2's page, the Agents page, Settings and the
builder, no page error or cut text. OSWorld2's limitations fit one
screen at 390 px, light and dark. `/ui-sweep --only osworld2` at
`fc0bdb1f`, this pull request's third commit before the rebases, dropped
since (it set `limits.run_again`, which main's `--retries` replaced): 6
screenshots, none with a page error or cut text.
- `PYTHONPATH=app python3 -m nullius standard check`: ok. `python3
app/tests/catalogue_docs.py --write`, then the check: up to date.
`python3 app/tests/field_contract.py`: ok. `app/site/readme_media.py
--check`: current. The docs check reports the same two findings in
`AGENTS.md` as on main.
- **Where the runs below ran.** The commits named here are of the port's
working branches, not in this repository:
- `c3637ffd` is this pull request's tree with main's two later commits
merged in;
- `f74d1a00` and `58c2b586` add one test commit: two JSON files, no
code. It gives OSWorld2's Original one more model, MiniMax M3.1 Flash,
the name it is served as, its pin to MiniMax and `--max_tokens 32768`.
That model is a test stand, not in this pull request;
  - `58c2b586` is this entry on the core before #124;
- `34e35370` and `8e2e381b` are of the port before its move to this
repository's history.
- **The controls,** recorded in `standard.json`'s `proven`:
- Offline, no services, on 2026-11-04, on an earlier commit: the 36 ids
are the same twice, and the null control scores 0 on 4 tasks;
- All tasks, on 2026-11-15, on `fc0bdb1f`, this pull request's third
commit before the rebases, dropped since (it set `limits.run_again`,
which main's `--retries` replaced); the next commit only writes this
record, and the one after changes three tests. The null control ran on
all 108 tasks, one launch with each task a shard of its own, at 16 MB/s
and 160 IOPS a VM, in six lanes with at most seven VMs at a time; it
took 1 h 32 min. 108 and 093 failed their setup under that load and
passed alone at 24 MB/s and 240 IOPS, in 24 more minutes. The grader
cost $0.14. `nullius standard prove osworld2 --dataset test_v2 --job
nop=<merged job>` passed: the 108 ids are the same twice, 0 of 108 at
full score, a mean of 0.0021 against a bound of at most 0.01. Five tasks
score above 0 on the untouched VM, as a limitation says;
- every task is in line with the same control on all 108 tasks on
`34e35370`, which had the same mean; 077's LLM grader gave 0.054 both
times.
- **GitLab made by Nullius,** on a server whose GitLab had been set up
by hand: the null control of 026, alone, made GitLab first (cloned,
configured, its root token, its runner, its snapshot) and scored 0.0 in
15 min; 041, alone, started from that snapshot and scored 0.0 in 7 min;
a snapshot of an older stamp was made anew by itself. No model call, no
GitLab left after either, and the token in no file of the jobs.
- **An agent's score on every kind of task,** each above the same task's
null control (0.0 in each):
- **Seven kinds,** on `58c2b586`, by the test stand's model in
OSWorld-V2's M3 agent, from the command line:
    - desktop apps, 045: 1.0;
    - mock websites, 052: 1.0;
    - Overleaf, 072: 0.775;
    - GitLab, 026: 0.5, a partial score;
- a proxy, 075: 1.0 (recorded 0.9999999999999999), with the unit's proxy
carrying new outbound connections while the agent acted;
    - the public internet, 027: 1.0;
    - the LLM grader, 003: 1.0, with both grader calls answered 200.

Each of the seven gives its recorded score again at `c3637ffd` when its
outcome is computed from its saved records, with no VM and no
evaluate().
- **The eighth, the simulated user:** 007, 0.8, by Muse Spark on
`8e2e381b`; the simulated user answered 7 times. It was not run on this
base.
- **Live on this base, against OpenRouter:**
- on `c3637ffd`, the null control on 087, a graded task: 0.0. The
grader's call went to Anthropic by the definition's pin, and
`standard.served` reads `held: true`;
- on `f74d1a00`, the test stand's model on 052: 1.0. All 68 calls went
to MiniMax by the test commit's pin and named it, `held: true`, with no
error.
- **This pull request's own agents** have no positive score on this
base. M3 ran one task here, 017, on the core before #124 (`90fde73e`),
and scored 0.0: the agent declared the task failed at its 120th step.
Muse Spark ran none. Their scores before the move, on `34e35370` and
earlier:
- M3 on 9 tasks: 070 1.0 twice, 030 1.0, 045 1.0, 013 1.0 then 0.0, 031
0.714, 017 0.429, 010 0.0 three times and once an environment error; 026
and 098 left no score (a full context window, a spend guard);
- Muse Spark on 3 tasks: 045 1.0, 007 0.8; 052 left no score (a launch
error of ours, then a spend guard).
- **Every attempt before the move,** two test stands' models included,
was recorded and triaged: 61 attempts on 33 tasks. 43 were scored (21
full, 11 partial, 11 zero), 10 ended in an environment error and 8 were
stopped by a guard or by hand. None was an agent error. Each of their
full and partial scores was checked for the false-success patterns the
port's audits found: none.
- Not run: a positive score of M3 or Muse Spark on this base; the
simulated user on this base; task 041 with an agent; a parity run of the
paper's 108 tasks; the 10 skipped tests with the upstream named; the
docker tier; the site build.

## Risk

- OSWorld2's driver and OSWorld-V2's scripts run only when someone runs
OSWorld2. The Agents page counts eight agent benchmarks, and the
generated agent pages gain an OSWorld2 column. No agent there declares
`agent.computer/v1`.
- **Scores.** A run compares with the paper's row by its model alone:
Docker instead of AWS, no US residential proxy, release `osworld-v2.1`,
the grader through OpenRouter, and by default 100 steps with thinking
off. A wrong error kind would move a unit between "environment error"
(left out of the adjusted rate) and the agent's error (counted).
- **Steps.** A Muse Spark run that sets no step limit now takes up to
500 steps, its wrapper's budget, where it took 100: a longer and dearer
episode. M3 keeps 100. A saved recipe or command that names a number
keeps it.
- **The machine.**
- VMs, the mock websites, GitLab and the proxies run on the machine the
units run on, outside a run's constraints in Settings; `-j` bounds them.
- GitLab's Docker in Docker and the websites' image build run as
privileged containers.
- In each of the driver's processes, `startup/sitecustomize.py` wraps
the creation of the VM's container: its network, its ports on 127.0.0.1,
its disk and its write limits. The VMs need a free Docker subnet.
- The VM's clock is UTC+8: a dated task set up before 16:00 UTC and
acted on after can score 0.
- The first unit of 026 or 041 on a machine clones Task-Web/gitlab at
its pin, pulls its images (about 7 GB) and makes its snapshot: about 8
minutes more, once.
- **Secrets.** GitLab's root token is made on the run's machine and kept
there in a file for its user alone, beside GitLab's clone; only the
processes of 026 and 041 get it. Model keys stay with the relay. Tasks
001, 006 and 010 start their mail client offline, unlike the upstream.
- Stacked on #133. Once it is merged, this branch is rebased onto main.
- Undone by reverting the commits; the catalogue entry goes with them.

---------

Co-authored-by: Oleg <247799477+sirotaoleg@users.noreply.github.com>
Co-authored-by: Etienne Filhol <etienne.filhol@hotmail.fr>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Standard: run each unit's agent in a Boat VM of its own

1 participant