Skip to content

refactor(judge): call the System One judgement Decision - #196

Merged
aitianshi merged 5 commits into
mainfrom
refactor/decision-judge
Nov 18, 2026
Merged

aitianshi merged 5 commits into
mainfrom
refactor/decision-judge

Conversation

@aitianshi

@aitianshi aitianshi commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #194

What

The first judgement is called "Decision" in the place of "System One", TypeSafe's product name: it needs a decision model, and OpenRouter and OpenAI serve them too.

  • Pages say "Decision model" (the trial's tab, the not-configured notes, the job page's button, Settings); code and docs say Decision. The protocol id systemone (TypeSafe's API path) stays.
  • Settings are DECISION_PROVIDER, DECISION_PROTOCOL, DECISION_MODEL. One saved under its old SYSTEM_ONE_* name is read as the new one, and the next save writes the new name.
  • Without a provider, the judgement says it needs a model, not the TypeSafe key.
  • Settings, Judges: the cards read "Decision model" and "LLM".
  • Stored judgements keep their folder (judgements/) and still load.

Touches

  • backend: judge/client.py, judge/store.py, judge/batch.py, settings.py, providers.py, onboarding.py
  • routes: web/judge.py, web/settings.py, web/traces.py
  • templates and static: Settings, the judge tabs, the job, compare and feasibility pages, settings/judges.js, settings/autosave.js
  • docs: settings/decision.md (was system-one.md), settings/README.md, judgement.md, llm-judge.md, model-providers.md, the README, the site's navigation
  • tests: judge, settings and route tests

Before and after

main and this branch, dark, 1280 px.

Settings, Judges

Before After
Before: System One and LLM judge After: Decision model and LLM

A trial's tabs

Before After
Before: System One judge tab After: Decision model tab

Verified

  • On main (9bc4b12): backend suite 2240 OK, route suite 562 OK.
  • Tests: an old SYSTEM_ONE_* setting is read as the new one and Settings does not list it as unread; a save writes only the new names; without a provider the judgement asks for a model.
  • No System One, system-one, SYSTEM_ONE or system_one left outside docs/releases/ and the migration's own names.
  • Not verified: a real call to a decision model (no key used).

Risk

  • A script or CI that sets SYSTEM_ONE_* keeps working; one that reads them back from Settings' file finds the new names after a save.
  • The form field and route names changed (/settings/decision/models): a page open from before this reloads.
  • Undone by a revert; a revert reads the new names as unknown, so a setting saved in between is set again.

Every page, message, document, setting and name says Decision: the
DECISION_PROVIDER, DECISION_PROTOCOL and DECISION_MODEL settings, the
decision lane, /settings/decision/models and docs/settings/decision.md.
Settings saved as SYSTEM_ONE_* are still read, and saving the new name
from the page takes the old line out of the file. Judgements stay in the
same folder. TypeSafe's protocol keeps its id and path, systemone.

Without a provider, the judge says it needs a model, not the TypeSafe
key: OpenRouter and OpenAI serve decision models too.
…e new one

The page showed DECISION_* unset and listed SYSTEM_ONE_* as unread while
the judge still used them. The file's old names are now read as the new
ones, so the page shows them set and the next save writes the new names.
The Judges section already says judge.
@github-actions

This comment has been minimized.

@github-actions

Copy link
Copy Markdown

🟢 PR judge · 8 of 8 yes · #196 ↔ #194

Merge risk 🟠 SIGNIFICANT the judge's reading of the diff
Priority 🟠 HIGH the deepest path is app/nullius/judge/__init__.py
Question Verdict
Changes do what the issue asks · 5 of 5 criteria ✅ yes
Issue and pull request describe the same change ✅ yes
Description matches the diff, no more, no less ✅ yes
Diff stays in scope ✅ yes
Tests cover the change, or Verified says how ✅ yes
Docs follow the change ✅ yes
Risk named in the description ✅ yes
Before and after shown · 4 images · the diff touches what is visible ✅ yes
Acceptance criteria, 5 of 5 met
# Criterion Verdict
1 Pages, settings, code and docs say "Decision" where they said "System One"; the protocol id systemone (TypeSafe's API… ✅ met (cut)
2 Settings are DECISION_PROVIDER, DECISION_PROTOCOL, DECISION_MODEL; one saved under its old SYSTEM_ONE_* name is… ✅ met (cut)
3 Without a provider, the judgement says it needs a model, not the TypeSafe key. ✅ met (cut)
4 Settings, Judges, names its cards "Decision model" and "LLM". ✅ met (cut)
5 Judgements already stored still load. ✅ met (cut)
Checks the script runs itself, 4 of 4 pass
No machine attribution trailer in the commits or the description ✓
New code under app/, docs/, .github/ or .claude/ ✓
No Flask import under the backend package ✓
No key-shaped string in the diff ✓

some of the diff was cut before the judge read it

@aitianshi
aitianshi merged commit b9ce577 into main Nov 18, 2026
2 checks passed
@aitianshi
aitianshi deleted the refactor/decision-judge branch November 18, 2026 00:16
aitianshi added a commit that referenced this pull request Nov 18, 2026
… last call took (#197)

Closes #195

## What

On a trial's judge tabs the Judge button moves to the left, under the
experimental badge. Once judged, a line beside it says what the
judgement took: "Judged in 12.40s · jev-latest · 1.2k tokens · $0.0031",
each part only when the record has it.

The client times its calls and keeps the seconds with the usage; a
judgement of several calls adds them up. A judgement stored before this
has no time and shows the rest.

## Touches

- backend: `judge/client.py` (the seconds in the usage, summed in
`merge`)
- templates and static: `traces/_judgement.html`, `style.css`
- docs: `settings/decision.md` (the record's `usage.seconds`)
- tests: `test_judge.py`, `test_judge_openrouter.py`,
`ui_tests/test_trial_head.py`

## Before and after

#196 and this branch on top of it, a trial with a stored judgement,
dark, 1280 px.

| Before | After |
| --- | --- |
| <img width="420" alt="Before: the button alone on the right"
src="https://github.com/user-attachments/assets/f15cd28d-47b9-4fab-8cb9-ea305026be17"
/> | <img width="420" alt="After: the button under the badge, what the
call took beside it"
src="https://github.com/user-attachments/assets/f29f6fd1-f683-439c-82f5-379f2652e7ed"
/> |

## Verified

- On #196 (`3483140`): backend suite 2240 OK, route suite 563 OK.
- A route test reads the line for a record with time, tokens and cost,
and for an older one with neither time nor cost. It caught a crash on
the older record, fixed before this was opened.
- Not verified: a real judge call's time (no key used; the stored record
is a seed).

## Risk

- Records written from now on carry `usage.seconds`; older readers
ignore it.
- Stacked on #196 (the Decision rename), which merges first: GitHub then
points this one at `main`.
- Undone by a revert.
aitianshi added a commit that referenced this pull request Nov 19, 2026
…ces on the machine and a null control (#134)

Stacked on #133: this pull request's own commits are the last sixteen.
Until #133 is merged, the diff shows its files too. It can be merged
after #133.

## What

OSWorld-V2's 108 tasks become a bundled benchmark, `osworld2`, on the
Nullius standard, at the upstream's release `osworld-v2.1`. Each task is
a unit in an Ubuntu VM of its own on the upstream's docker provider, set
up by the task's own class and scored from 0 to 1 by its own evaluate().
The tasks and their assets are gated on Hugging Face. Nullius does not
fetch them, and the entry is `gated` (#133): a unit's failure from a
model endpoint says its status alone, and the driver names an upstream
exception by a known class's name at most, never by its words. The
Original setup runs one of OSWorld-V2's own agents, M3 or Muse Spark,
chosen by the Agent option, from its published script, with every model
call sent to the relay. Claude Sonnet 4.6 is the LLM grader of 21 tasks
and the simulated user of 6. No provider is pinned: Served by is the
user's, as for OSWorld1. Only M3's runs count as comparable with the
paper's row. Its Review row and its generated page name the setup
"OSWorld-V2 agents". A null control runs one `DONE` action, then the
task's evaluate() on the untouched VM. The entry declares
`agent.computer/v1`. No agent from the Agents page runs it yet. Its
texts are one Good to know line and 9 one-line limitations.

The mock websites of 40 tasks are OSWorld-web at the same tag. The
guide's steps build them from source and serve them on the machine the
units run on. A unit that needs them is refused before its VM starts
when they do not answer. For 026 and 041, Nullius makes GitLab on the
run's machine before the first of them, with a root token it keeps
there, then starts it anew from its snapshot for each, one task at a
time; Overleaf's 2 tasks get Overleaf started anew. Each of the 8 tasks
that ask for a proxy gets a proxy of its own. The options are the agent,
by name, Thinking for M3, and disk write limits for each VM. The website
host suffix is asked on the Dataset step, for the four datasets with
website or GitLab tasks. Steps per task is with the limits, each agent's
own by default, as its wrapper runs it: 100 for M3, 500 for Muse Spark.
The VM provider, Docker alone, is on the Where step. By default M3 takes
100 steps with thinking off, as the upstream's M3 script does; the paper
ran M3 at 500 steps with thinking on. Muse Spark runs as published at
this release, while its published score came from newer upstream code; a
limitation says it does not compare, and a docs page says what differs.
There are six datasets: All tasks (108, the upstream's `test_v2.json`),
Offline (82), and four by the services their tasks need. Tasks 103 and
104 are not supported: their evaluate() needs FreeCAD on the driver's
machine. Review and the run's start ask for Docker and for the scoring
programs (ImageMagick, ffmpeg, tesseract) on the driver's machine. As
for OSWorld1, the Where step greys out the container card under the
Original, whose docker provider needs Docker on the machine. A guide,
`docs/benchmarks/osworld2-machine.md`, prepares a machine, runs the null
control on one task as its last setup step, and says what a run takes in
time and money. A second page, `docs/benchmarks/osworld2-results.md`,
says how a run differs from the paper and the upstream, and how a unit
is scored, stopped or left unscored.

## Touches

standard (an option's example and a machine setup's label: the
definition check, the benchmark and agent pages, the builder, Review) ·
catalogue · templates and static (the mark) · docs and site
(`docs/benchmarks/`, the generated pages, `AGENTS.md`) · routes (tests
only) · packaging (the entry's `.yml`, `.Dockerfile` and `startup/*.py`
as package data in `pyproject.toml`)

## Before and after

**Before:** The Benchmarks page before this change

![Benchmarks page
before](https://github.com/user-attachments/assets/718bb2be-441a-48e3-a0d6-dccea074f984)

**After:** OSWorld2 listed, with its mark

![Benchmarks page
after](https://github.com/user-attachments/assets/60e3c777-dd18-44de-b755-36e60fe3b0f1)

**Before:** The builder's Benchmark step under Computer use: OSWorld1,
beside the selected GAIA

![Benchmark step
before](https://github.com/user-attachments/assets/dfad41fe-fb70-4662-bd2f-56ee25df010d)

**After:** OSWorld1 and OSWorld2

![Benchmark step
after](https://github.com/user-attachments/assets/e122bfcf-ff51-4028-91c4-669330738459)

**Before:** The Agents page: "n/7 benchmarks it runs"

![Agents page
before](https://github.com/user-attachments/assets/808c7fba-cd24-48d9-b52d-71ef90717a09)

**After:** "n/8": OSWorld2 counts, and no agent runs it

![Agents page
after](https://github.com/user-attachments/assets/e1ae840a-c60f-4050-8ef4-3c4ac594750c)

**Before:** An agent's page, Codex: the benchmarks it runs

![Codex's page
before](https://github.com/user-attachments/assets/5a435d9b-e15b-49d0-9bb1-63706f1e1d18)

**After:** OSWorld2 listed: it needs `agent.computer/v1`, which Codex
does not offer

![Codex's page
after](https://github.com/user-attachments/assets/a07ef2b2-82e0-4284-8ce1-05de885ca1bd)

**New:** OSWorld2's page: its setup and its proofs by dataset

![OSWorld2's
page](https://github.com/user-attachments/assets/9ee0d906-9bad-4b34-a8fc-961ab9f95a63)

**New:** Its options: the agent by name, steps, thinking, the VM
provider, the website host suffix and the disk write limits

![OSWorld2's
options](https://github.com/user-attachments/assets/8f3dbf48-e642-4312-a22d-53ebbfb3d56a)

**New:** Its dataset and its Good to know line

![OSWorld2's
dataset](https://github.com/user-attachments/assets/5d94c401-9cd3-4df9-9a25-6d75cff43ca6)

**New:** Its limitations

![OSWorld2's
limitations](https://github.com/user-attachments/assets/7559fbcc-3a75-42aa-877c-7dd8a3d793af)

**New:** Dataset step: six datasets with their controls; All tasks
chosen asks for the website host suffix, and for no token

![Dataset
step](https://github.com/user-attachments/assets/c950ee7d-4fa1-4b4a-9f19-db24ad4c57fb)

**New:** Agent step: OSWorld-V2 agents; the Agents page's agents lack
computer use; the settings every agent shares, the disk write limits

![Agent
step](https://github.com/user-attachments/assets/0a10ee0a-c924-4fac-a156-1e65c2dc25c8)

**New:** The Original's own settings: MiniMax M3 or Muse Spark by name,
and thinking

![Agent step, the Original's
settings](https://github.com/user-attachments/assets/1fe7b5d1-780c-44e4-baa7-6e58a57a2400)

**New:** Muse Spark chosen: Thinking, an M3 setting, is gone

![Agent step with Muse
Spark](https://github.com/user-attachments/assets/72910ec5-29ab-458e-b737-029f5b60f174)

**New:** Model step, Like the paper: MiniMax-M3 on OpenRouter, Claude
Sonnet 4.6 as the judge, each served by any provider until one is named

![Model
step](https://github.com/user-attachments/assets/2e3db9e9-53f3-4586-8fc0-9c481a816b47)

**New:** Limits step: Steps per task, each agent's own by default, 100
for MiniMax M3

![Limits
step](https://github.com/user-attachments/assets/cd84ad6a-de7b-4c31-8d9c-7f7d645ce7f8)

**New:** Where step: the VM provider, Docker alone; no container or Boat
backend here; Docker and the scoring programs checked on this machine

![Where
step](https://github.com/user-attachments/assets/86a5af3a-ae54-412d-9a9b-8703e75c4d06)

**New:** Review: every decision, both prerequisites ready, no model key
in this fresh profile

![Review
step](https://github.com/user-attachments/assets/459653db-56c0-4aa7-a394-cbab9d5fbd56)

**New:** The null control's merged job on test_v2, its first rows

![Null control
job](https://github.com/user-attachments/assets/7d5338e5-1a0b-40d7-86ad-008eba176607)

**New:** 052 by the test stand's model in M3: 1.0

![Job of
052](https://github.com/user-attachments/assets/097f88a5-ee73-4cbb-ac11-cb5655e749ce)

Taken at 1400 px, light theme, on a fresh profile. Cut to part of their
page: the Benchmark step and Agents pairs (the top 1000 px), Codex's
page (the top 1100 px), OSWorld2's page (four parts) and the null
control's job (its first rows). The two jobs are runs named under
Verified, opened in the app. The Model step, Review, OSWorld2's options,
dataset and limitations, and the Dataset, Agent and Where steps were
taken again after main's Served by (#124), Boat (#150) and time limits
(#191), the shorter texts and GitLab made by Nullius, and checked at
`273cca5e`: the other pictures differ there only in the sidebar, which
they leave out. The two jobs were taken again at `c4427915`, on main at
`0d83e393`, whose job page names the judge's button anew (#196); the
other pages and steps pictured are the same there. OSWorld2's options
and limitations and the Dataset, Agent, Model, Limits, Where and Review
steps were taken again at `9526a5b1`, after the agents' names, the steps
as a limit, the VM provider on Where, the website host suffix on Dataset
and Served by with no default.

## Verified

- At `9526a5b1`, on #133 (`e768c632`) on main at `db6e8f96`, on Python
3.12: backend suite (`python3 app/tests/parallel.py`), 2688 tests in 144
modules, OK; route suite with Node on PATH, 643 tests in 53 modules, OK;
the wheel's check (`app/packaging/verify.py`), OK.
- **OSWorld2's 15 test modules,** 375 tests, OK; they need the entry, so
none passes without it. 10 are skipped here: they run only when the
environment names the upstream with its gated task classes, OSWorld-web
or GitLab. Three route test modules and three backend test modules
changed for the new entry: the benchmark lists and counts, the Where
step's container card, Settings' custom keys, and OSWorld2's case of
gated tasks (`test_standard_gated.py`). Option examples and setup labels
change `test_standard_options.py`, `test_prerequisites.py` and
`test_standard_bench_options.py`, and the route modules
`test_catalogue_routes.py`, `test_agents.py` and
`test_builder_where.py`.
- At `9526a5b1`, in a headless Chromium, light and dark, 1400 and 390
px: the builder's Agent step (the agents by name, Steps no longer
there), Model step (Any of your choice: no provider filled in), Limits
step (Steps per task, 100 for M3 and 500 for Muse Spark), Where step (VM
provider docker, fixed) and Review; no page error.
- In a headless Chromium, light and dark, 1400 and 390 px: the
Benchmarks page, OSWorld2's page, the Agents page, Settings and the
builder, no page error or cut text. OSWorld2's limitations fit one
screen at 390 px, light and dark. `/ui-sweep --only osworld2` at
`fc0bdb1f`, this pull request's third commit before the rebases, dropped
since (it set `limits.run_again`, which main's `--retries` replaced): 6
screenshots, none with a page error or cut text.
- `PYTHONPATH=app python3 -m nullius standard check`: ok. `python3
app/tests/catalogue_docs.py --write`, then the check: up to date.
`python3 app/tests/field_contract.py`: ok. `app/site/readme_media.py
--check`: current. The docs check reports the same two findings in
`AGENTS.md` as on main.
- **Where the runs below ran.** The commits named here are of the port's
working branches, not in this repository:
- `c3637ffd` is this pull request's tree with main's two later commits
merged in;
- `f74d1a00` and `58c2b586` add one test commit: two JSON files, no
code. It gives OSWorld2's Original one more model, MiniMax M3.1 Flash,
the name it is served as, its pin to MiniMax and `--max_tokens 32768`.
That model is a test stand, not in this pull request;
  - `58c2b586` is this entry on the core before #124;
- `34e35370` and `8e2e381b` are of the port before its move to this
repository's history.
- **The controls,** recorded in `standard.json`'s `proven`:
- Offline, no services, on 2026-11-04, on an earlier commit: the 36 ids
are the same twice, and the null control scores 0 on 4 tasks;
- All tasks, on 2026-11-15, on `fc0bdb1f`, this pull request's third
commit before the rebases, dropped since (it set `limits.run_again`,
which main's `--retries` replaced); the next commit only writes this
record, and the one after changes three tests. The null control ran on
all 108 tasks, one launch with each task a shard of its own, at 16 MB/s
and 160 IOPS a VM, in six lanes with at most seven VMs at a time; it
took 1 h 32 min. 108 and 093 failed their setup under that load and
passed alone at 24 MB/s and 240 IOPS, in 24 more minutes. The grader
cost $0.14. `nullius standard prove osworld2 --dataset test_v2 --job
nop=<merged job>` passed: the 108 ids are the same twice, 0 of 108 at
full score, a mean of 0.0021 against a bound of at most 0.01. Five tasks
score above 0 on the untouched VM, as a limitation says;
- every task is in line with the same control on all 108 tasks on
`34e35370`, which had the same mean; 077's LLM grader gave 0.054 both
times.
- **GitLab made by Nullius,** on a server whose GitLab had been set up
by hand: the null control of 026, alone, made GitLab first (cloned,
configured, its root token, its runner, its snapshot) and scored 0.0 in
15 min; 041, alone, started from that snapshot and scored 0.0 in 7 min;
a snapshot of an older stamp was made anew by itself. No model call, no
GitLab left after either, and the token in no file of the jobs.
- **An agent's score on every kind of task,** each above the same task's
null control (0.0 in each):
- **Seven kinds,** on `58c2b586`, by the test stand's model in
OSWorld-V2's M3 agent, from the command line:
    - desktop apps, 045: 1.0;
    - mock websites, 052: 1.0;
    - Overleaf, 072: 0.775;
    - GitLab, 026: 0.5, a partial score;
- a proxy, 075: 1.0 (recorded 0.9999999999999999), with the unit's proxy
carrying new outbound connections while the agent acted;
    - the public internet, 027: 1.0;
    - the LLM grader, 003: 1.0, with both grader calls answered 200.

Each of the seven gives its recorded score again at `c3637ffd` when its
outcome is computed from its saved records, with no VM and no
evaluate().
- **The eighth, the simulated user:** 007, 0.8, by Muse Spark on
`8e2e381b`; the simulated user answered 7 times. It was not run on this
base.
- **Live on this base, against OpenRouter:**
- on `c3637ffd`, the null control on 087, a graded task: 0.0. The
grader's call went to Anthropic by the definition's pin, and
`standard.served` reads `held: true`;
- on `f74d1a00`, the test stand's model on 052: 1.0. All 68 calls went
to MiniMax by the test commit's pin and named it, `held: true`, with no
error.
- **This pull request's own agents** have no positive score on this
base. M3 ran one task here, 017, on the core before #124 (`90fde73e`),
and scored 0.0: the agent declared the task failed at its 120th step.
Muse Spark ran none. Their scores before the move, on `34e35370` and
earlier:
- M3 on 9 tasks: 070 1.0 twice, 030 1.0, 045 1.0, 013 1.0 then 0.0, 031
0.714, 017 0.429, 010 0.0 three times and once an environment error; 026
and 098 left no score (a full context window, a spend guard);
- Muse Spark on 3 tasks: 045 1.0, 007 0.8; 052 left no score (a launch
error of ours, then a spend guard).
- **Every attempt before the move,** two test stands' models included,
was recorded and triaged: 61 attempts on 33 tasks. 43 were scored (21
full, 11 partial, 11 zero), 10 ended in an environment error and 8 were
stopped by a guard or by hand. None was an agent error. Each of their
full and partial scores was checked for the false-success patterns the
port's audits found: none.
- Not run: a positive score of M3 or Muse Spark on this base; the
simulated user on this base; task 041 with an agent; a parity run of the
paper's 108 tasks; the 10 skipped tests with the upstream named; the
docker tier; the site build.

## Risk

- OSWorld2's driver and OSWorld-V2's scripts run only when someone runs
OSWorld2. The Agents page counts eight agent benchmarks, and the
generated agent pages gain an OSWorld2 column. No agent there declares
`agent.computer/v1`.
- **Scores.** A run compares with the paper's row by its model alone:
Docker instead of AWS, no US residential proxy, release `osworld-v2.1`,
the grader through OpenRouter, and by default 100 steps with thinking
off. A wrong error kind would move a unit between "environment error"
(left out of the adjusted rate) and the agent's error (counted).
- **Steps.** A Muse Spark run that sets no step limit now takes up to
500 steps, its wrapper's budget, where it took 100: a longer and dearer
episode. M3 keeps 100. A saved recipe or command that names a number
keeps it.
- **The machine.**
- VMs, the mock websites, GitLab and the proxies run on the machine the
units run on, outside a run's constraints in Settings; `-j` bounds them.
- GitLab's Docker in Docker and the websites' image build run as
privileged containers.
- In each of the driver's processes, `startup/sitecustomize.py` wraps
the creation of the VM's container: its network, its ports on 127.0.0.1,
its disk and its write limits. The VMs need a free Docker subnet.
- The VM's clock is UTC+8: a dated task set up before 16:00 UTC and
acted on after can score 0.
- The first unit of 026 or 041 on a machine clones Task-Web/gitlab at
its pin, pulls its images (about 7 GB) and makes its snapshot: about 8
minutes more, once.
- **Secrets.** GitLab's root token is made on the run's machine and kept
there in a file for its user alone, beside GitLab's clone; only the
processes of 026 and 041 get it. Model keys stay with the relay. Tasks
001, 006 and 010 start their mail client offline, unlike the upstream.
- Stacked on #133. Once it is merged, this branch is rebased onto main.
- Undone by reverting the commits; the catalogue entry goes with them.

---------

Co-authored-by: Oleg <247799477+sirotaoleg@users.noreply.github.com>
Co-authored-by: Etienne Filhol <etienne.filhol@hotmail.fr>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Judge: the first judgement is named for one vendor's product, though any decision model answers it

1 participant