Skip to content

feat(runs): fold a launch's shards into one row per participant - #137

Merged
harenome merged 1 commit into
ThinkFlowLab:mainfrom
harenome:feat/launch-fold-shards
Nov 16, 2026
Merged

harenome merged 1 commit into
ThinkFlowLab:mainfrom
harenome:feat/launch-fold-shards

Conversation

@harenome

@harenome harenome commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #136

What

On Runs → By launch, an agent or control with 2 or more shards is now one row, when cli merge would merge those shards. The row shows the numbers of the merged job. launches.fold makes that job in memory: the trials of every shard, and the job record from merge.result_of. result_of is the code that merge used inline to compute the job record, now a shared function. merge.check keeps the trial records that it already reads.

  • The rate is over every trial with a verdict, never an average of shard rates.
  • Units that no shard ran come from merge.check and stand beside the count ("2 of 8 not run"). They are never misses. The notes of merge are the tooltip.
  • The state is the most telling state of the shards, with a warning such as "1 of 2 shards stopped".
  • The header gives the dataset size, not the size of one shard.

The row is folded by default. A click on the row, or Enter or Space on its chevron, unfolds one row per shard ("shard 2 · host-2"), each with its Open. The fold uses the shared chevron and slide of disclosures.js, and respects reduced motion. A refresh of the list keeps an unfolded row unfolded. Controls fold the same way, stay unranked and stay out of Compare. The task-check line is judged over every shard: "Task checks incomplete: oracle not run on shard 2."

A participant with one shard keeps its plain row. Shards that merge refuses also keep their rows. Job pages, cli merge and merged jobs do not change. The header counts every shard ("8 runs"), every shard has its row, and each folded row says how many shards it holds ("2 shards").

A fold reads every trial record of its shards. It is kept in memory for the shard jobs that the workspace cache gives, and it is read again only when the workspace parses a shard again. A new set of shards replaces each kept fold that shares a shard with it, so the cache holds one fold per shard at most.

Touches

backend · the standard (merge.py) · routes · templates and static · docs and site

Before and after

One launch over 2 machines, 8 units: 2 agents, nop and oracle. Shard 2 stopped while terminus-2 ran (2 of its 4 units), and the oracle did not run on shard 2. Light theme, 1280 px.

Before After, folded
1-before-shards-apart 2-after-folded

Before: 8 rows, 2 rates for each agent, "4 tasks", "oracle not run". After: 4 rows, "8 tasks". terminus-2 shows 50% and "3/6 · 2 of 8 not run" with "1 of 2 shards stopped". The task-check line reads "Task checks incomplete: oracle not run on shard 2."

After, unfolded (click and keyboard) After, dark, 390 px
3-after-unfolded 4-after-dark-390

Verified

  • Backend test Fold.test_a_fold_reads_as_the_merged_job: three shards, one absent, and one with a planned unit that did not run. The folded job equals parse_job of the merge.merge output on every field the row shows (solved, rates, outcomes, planned, errors, start, end, time, tokens, cost, mean score, budget). The rate is not the mean of the shard rates. The units not run and the notes are those of merge. Shards that merge refuses are not folded.
  • Backend test Fold.test_a_fold_is_read_again_only_when_the_workspace_parses_a_shard_again: a second fold of the same cached jobs does not call merge.check. After a trial record changes, it calls merge.check again. After a third shard appears, the cache holds only the fold of the three.
  • Route tests test_launch_shards: 8 shards give "8 runs", 8 shard rows and 4 folded rows that each say "2 shards". The rows are folded by default. Each shard row has its Open, and the folded row has none. The rate and time of the folded row equal those of the merged job. "2 of 8 not run" and "1 of 2 shards stopped" show. The controls fold and stay out of Compare. A nop that passes one task on shard 2 gives "A control did not behave". A stopped nop shard gives "nop stopped on shard 2". Participants with one shard keep their plain rows.
  • On c7af06d5: backend suite python3 app/tests/parallel.py (1989 tests) and route suite python3 app/tests/parallel.py app/ui_tests (328 tests) pass in a clean container.
  • Browser, with the fixture above: folded and unfolded, light and dark, 1280 and 390 px. No page errors and no horizontal page scroll. A click on the row unfolds it, and the shard rows slide open and closed. Tab moves the focus to the chevron of the oracle, Enter unfolds it, and Space folds it again. With reduced motion, the rows show at once. At 390 px, a long agent name wraps and the chevron stays on the line of the name. The name, its "2 shards" line and the shard rows start at the same left edge.
  • Render time of /runs?view=launch, median of 15 requests after a first one, with the same parsed jobs given to the page. 2 agents and nop and oracle, each 4 shards of 500 units (8000 trials): 71 ms before this change, 101 ms after. The first render reads the trial records once: 540 ms, against 199 ms before. The rest of the added time is the outcome bars of the 4 folded rows, over 2000 trials each. 8 shards of 1 unit each (32 trials): 11 ms before, 13 ms after.

Risk

If the fold is wrong, a row shows numbers that differ from the merged job. The backend test compares the two directly. merge writes the same result.json as before through result_of, and its tests pass unchanged. The first render of a large sharded launch reads all its trial records once (+0.3 s for 8000 trials). Later renders use the cache until a shard changes. The cache holds one fold per shard at most, with the parsed jobs that the workspace cache also holds. Open pull requests #124, #133 and #134 also change merge.py: the second to merge must rebase. Revert the commit to undo.

A launch run over several machines drew one row per shard: an agent on
two machines was two rows, each with its own rate, and its shards' rows
were mixed with the other agents' in the ranking.

Fold each agent's or control's shards into one row when it has two or
more and `cli merge` would merge them. The row shows the merged job's
numbers: the trials of every shard, and the job record from
`merge.result_of`, now shared with `merge`. The rate is over every trial
with a verdict. Units that no shard ran stand beside the count ("2 of 8
not run"), never among the misses. The state is the most telling one of
the shards, with a note such as "1 of 2 shards stopped".

The row unfolds into one row per shard, named by shard and machine, each
with its Open. Controls fold the same way and stay unranked. The
task-check line is judged over every shard: "oracle not run on shard 2".

A fold reads every trial record of its shards. It is kept for the shard
jobs the workspace cache hands out, and read again only when the
workspace parses a shard again. A new set of shards replaces each
kept fold that shares a shard with it, so the cache holds one fold per
shard at most.
@harenome
harenome merged commit e7c3e85 into ThinkFlowLab:main Nov 16, 2026
2 checks passed
@harenome
harenome deleted the feat/launch-fold-shards branch November 16, 2026 10:18
aitianshi added a commit that referenced this pull request Nov 16, 2026
No linked issue: the owner asked for v0.1.1 as a release, so that the
Hugging Face endpoint setting (#129) reaches PyPI.

## What

Prepares v0.1.1. The version becomes `0.1.1`, and
`docs/releases/v0.1.1.md` gives the highlights since v0.1.0, a
Contributors section, the full changelog of every pull request in
`v0.1.0..main` (#114, #117, #120, #122, #125, #127, #129, #132, #137,
and #135 among the test and CI changes) and the installation notes, in
the shape of v0.1.0's. Install examples, the release guide, the docs'
release-notes link and menu, and the workflows' default tags name 0.1.1.
The release tooling already accepts final `vX.Y.Z` tags and does not
change.

## Touches

CI and skills (the release and publish workflows' default tags) · docs
and site

## Verified

- Backend suite (`python3 app/tests/parallel.py`): 1991 tests in 116
modules, OK. Route suite: 428 tests in 41 modules, OK.
- `catalogue_docs.py --write` and `diagrams.py --write` left no change.
- `python3 app/packaging/release.py build --tag v0.1.1` built and
verified `nullius_eval-0.1.1-py3-none-any.whl`,
`nullius_eval-0.1.1.tar.gz` and `SHA256SUMS`; `release.py verify --tag
v0.1.1` and `verify.py` on the wheel passed, including the
installed-tool and profile-preservation checks.
- The workflows have not run; they run only by hand on `main`.

## Risk

Version strings and docs only. A wrong default tag in a workflow is an
input the person running it can change, and `release.py` refuses a tag
that does not match `pyproject.toml` or has no notes. Nothing is tagged,
drafted or published by this change. Undo with a revert.
aitianshi pushed a commit that referenced this pull request Nov 19, 2026
…viders, set its own limits and prove a merged null control (#133)

## What

A benchmark of the standard can now say more about how its units run and
end, which a computer-use benchmark like OSWorld2 (#134) needs. `gated`,
the new limits and `prove`'s new options are opt-in. The relay's rules
and the rules of how a unit ends change for every benchmark of the
standard; Risk lists them.

- **Gated tasks** (`gated: true`): a unit's failure that the relay's
records explain, a model endpoint's (the agent's, the judge's or the
simulated user's), says in its `result.json` the status and class alone,
never the provider's words, which can quote the request. The trial's
call records keep the provider's words, and an error a driver raises
itself is kept as the driver words it.
- **A benchmark option's values by name** (`names`: {value: name}): the
builder's cards and select, Review and the benchmark's page show the
name, `about` stays its tooltip, and a recipe and the CLI keep the
value. OSWorld1's agents are named as their upstream spells them.
- **An option on its own step** (`step`): `where`, where the work runs,
on the Where step, one value as fixed, and in Review's Where row;
`dataset`, what some datasets need, on the Dataset step for those
datasets. OSWorld1's VM provider is on Where.
- **Limits of a benchmark's own:**
- `limits.score_seconds`: how long `score` may take, the benchmark's
level below `--timeout score=` (#191); the Limits step shows it as the
benchmark's;
- `limits.call_seconds`: how long the relay waits for one model call,
also `relay serve --timeout`;
- **How a unit ends:**
- a driver's error that begins `EnvironmentSetupError: `,
`VerifierError: `, `CancelledError: ` or `CauseUnknownError: ` keeps
that kind;
- the agent's own last call refused as longer than the model's context
window is `ContextWindowExceededError`, the agent's;
- a last 429 is read from the agent's own calls in the order they began,
so a side model's later 200 does not cover it;
- a unit a guard's stop ended keeps the score its driver gave, its own
last call's 429 notwithstanding, unless that 429 said the credit is
spent. `score` can say the stop's reason (`verifier_result.stopped`);
- with #154's `--retries`, a unit a guard asked to stop and a full
context window are not tried again;
- for an upstream's own answerer: `OutputLengthExceededError` only when
its last reply has neither text nor a tool call. When the answerer fails
and its last reply was a Responses object answered 200 and `failed`,
that is an API error.
- **The relay:**
- a non-streamed 200 to a model call with no reply in it reaches the
caller and the record as a 502, or the 429 or 402 of an error it held
(the record keeps `answered: 200`). #124 leaves counting an error inside
a 200 to a pull request of its own; this one takes only a 200 with no
reply in it;
- a stream cut off midway, after the caller had its 200, is recorded as
a 502, and its booking against the spending cap stays whole;
- a call refused with 400, 413 or 422 records each inline image it sent
by its declared type, the type its bytes show and its size, never a
byte;
- a `stop` file beside the relay's records: the relay notes in it an
agent's call refused for the model's context window, and that note
refuses nothing. Once a benchmark's guard writes a line of its own
there, the agent's later calls are refused unsent, while side models and
the judge's and simulated user's routes go on;
- a failed Responses object with no usage records a cost of 0, and its
booking is kept.
- **A job's name:**
- a run holds its job name until its process ends, and one under a name
another run still holds is refused before it writes anything;
- a rerun into the same job moves each rerun trial's earlier records
(`.work/relay`, `result.json`, `verifier/`, `agent/`) aside to
`.work/earlier.<ns>`.
- **prove and merge:**
- `nullius standard prove ... --bench-option NAME=VALUE` sets the
benchmark's own options for the units check and the controls;
- `--job nop=MERGED` (or `oracle=`) checks a merged launch of a control
against its bound instead of running it, and every trial's own record;
- `merge` accepts shards that differ in benchmark options and keeps each
shard's in `merged_from`;
- `merge` refuses, unless `--force`, a trial whose record names another
job, benchmark, version, control or agent, and a job itself merged
despite differences. It lists the shards absent from a merge only when
the launch has at most 1,000.
- **Smaller:**
  - a trace that cannot be built leaves `agent/trace.error.txt`;
  - an Original's trajectory keeps its tool calls;
  - each call's record keeps the request's sha256;
  - `thinking_tokens` count as reasoning;
- a setup's `comparable` can list the models whose runs compare with the
paper, and the builder's message for another model says so;
- a test checks that every file a host's snapshot takes of a bundled
entry is package data.

**Read first:** `failure`, then `_hollow` and the stop in
`Relay._forward` (`app/nullius/standard/relay.py`). Next, the outcome
rules in `Run` (`app/nullius/standard/run.py`), then `names` and `step`
in `recipe.compile` and `benchOptions` in the builder, the `--job` check
in `prove.py` and the trial check in `merge.py`. Then
`test_standard_gated.py`, `test_standard_context_window.py` and
`test_standard_controls.py`. The rest is the definitions' checks, the
CLI options, the sidecar's argument and the pages.

## Touches

the standard (`app/nullius/standard/`: `relay`, `run`, `prove`,
`definitions`, `merge`, `job`, `cli`, `container`, `recipe`, `fidelity`,
`retries`) · the builder (`app/static/builder/`, `app/web/builder.py`:
the score's default on the Limits step, an option's names, and an option
on its own step) · the benchmark page (an option's names) · OSWorld1's
definition (its agents' names, its VM provider on Where) · docs and site
(`docs/standard.md`, `docs/cli/run.md`, `docs/cli/read-jobs.md`,
`docs/builder.md`, `docs/settings/model-providers.md`)

## Verified

- At `e768c632`, on main at `db6e8f96`, on Python 3.12: backend suite
(`python3 app/tests/parallel.py`), 2310 tests in 129 modules, OK; route
suite with Node on PATH, 639 tests in 53 modules, OK; the wheel's check
(`app/packaging/verify.py`), OK.
- **An option's names and its step, in a headless Chromium, light and
dark, 1400 and 390 px,** with #134 on top: OSWorld1's agents by name in
their select, OSWorld2's in their cards; the VM provider of each as a
fixed value on the Where step, shown only while a participant that takes
it is chosen; OSWorld2's Website host suffix on the Dataset step for All
tasks, not for Offline, no services; Served by empty on any choice of
model. No page error.
- **The 12 changed test modules on main's source** (`5bcfd82d`): 9 fail,
2 pass, and 1 does not end.
- `test_standard_context_window`: 19 of 20; `test_standard_controls`: 17
of 43; `test_standard_relay`: 8 of 58; `test_standard_gated`: 4 of 6.
- `test_standard_run`: 2 of 50; `test_standard_trace`: 2 of 20;
`test_standard_parts`, `test_standard_served_by` and
`test_standard_units`: 1 each.
- `test_standard_shard` does not end within 4 minutes: one of its merges
names 2^64 shards, and main's `merge` lists every absent one.
  - `test_standard_versions` and `test_dependency_surface` pass on both.
- **Covered in #134, not here:** that `limits.call_seconds` reaches the
relay and the sidecar's `--timeout`. OSWorld2 is the first entry to set
it (`test_osworld2_muse.py`).
- **#124's own tests.** Three were changed for two deliberate
differences: a 200 with no reply is a 502, and a failed Responses object
is read as failed. The rest of `test_standard_served_by` passes as main
has it; `write_shard` in `test_standard_shard` takes #137's `reward`
beside this branch's `bench_options`.
- `PYTHONPATH=app python3 -m nullius standard check`: ok. `python3
app/tests/catalogue_docs.py --write`, then the check: up to date.
`python3 app/tests/field_contract.py`: ok. `app/site/readme_media.py
--check`: current. The docs check reports the same two findings in
`AGENTS.md` as on main.
- Not run: a benchmark already on main, live, on this branch; the
container backend with `--timeout`; the docker tier; the site build.

## Risk

- **For every benchmark of the standard:**
- a non-streamed 200 with no reply is a 502, and a stream cut off midway
is recorded as one;
- the agent's own last call refused for its context window is the
agent's error (`ContextWindowExceededError`), where it was an API error
outside its rate;
- an agent's last 429 followed by a side model's 200 makes the unit
`rate_limited`, where its score stood;
- an upstream answerer whose last reply was a `failed` Responses object
is an API error, where it was the agent's;
- a driver's error of a kind it names: a setup error or a stop is left
out of the agent's rate, and a verifier error or one of no known cause
counts as unknown. A wrong kind would move a unit between them;
- `prove` runs no control when the units check failed, and fails a
control job with a unit that has neither a score nor an error;
- a run under a job name another run still holds is refused, and a rerun
into an existing job moves the rerun trials' records aside;
- `merge` refuses a trial whose record names another job unless
`--force`.
- **Records.** Stored records gain fields and keep the old ones.
- These commits sit on main at `db6e8f96`, with #124, #150, #191, #154,
#174 and the Recipes page (#205). #124's `--served-by` code is main's as
merged; #191's time limits come first, so `--timeout score=` is used
over `limits.score_seconds`, and a unit's cap on model calls
(`UnitCallCapError`) is read before this PR's outcome rules. #154's
`--retries` replaces the one more run of a rate-limited unit, so the
earlier `limits.run_again: false` that switched it off is gone; main
counts a provider's 402 as a usage limit, and the tests follow. The
first and fourth commits gave a definition pins of its serving
providers, defaults of Served by after review; the last commit removes
them, as the reviewer asked, since no benchmark declares one: the diff
over main has none, and Served by is the user's alone, as in #124.
- The fifth commit shortens this PR's docs and messages. No logic
changes.
- **Names and Where.** An option's names change only what is shown; a
recipe saved before keeps working. An option of where the work runs
leaves the Agent step for Where, and one some datasets need is asked on
Dataset: OSWorld1's VM provider is now on Where.
- Undone by reverting the commits.

---------

Co-authored-by: Oleg <247799477+sirotaoleg@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Runs: fold a launch's shards into one row per agent or control

1 participant