Skip to content

feat(builder): add a Recipes page for the saved recipes - #205

Merged
harenome merged 5 commits into
ThinkFlowLab:mainfrom
harenome:feat/recipes-page
Nov 18, 2026
Merged

harenome merged 5 commits into
ThinkFlowLab:mainfrom
harenome:feat/recipes-page

Conversation

@harenome

@harenome harenome commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #204

What

A new Recipes page (/recipes) shows the saved recipes in a table. The sidebar link is in the Manage group, with a new i-recipe icon.

  • Table. The page uses the table toolkit (_tables.html table, tables.js) with search, Columns, Sort by and the pager, as Launches and Workspaces do. The columns are the name, one column for each builder step from 01 to 07 (Benchmark, Runner, Dataset, Agent, Model, Limits, Where), and Saved. A step cell holds the values of that step's Review rows. A long value is cut with an ellipsis, and the tooltip shows the full value.
  • Check. The page does not add a check. For a Harbor recipe, recipes.js calls the recipe core: fromJSON, review, validate and providerBlocker. For a standard recipe, the route runs the same compile as the builder's Review step (/new-run/runner), now one helper (_standard_compile). A row with an error shows "Cannot run here" with an alert icon. The tooltip gives each reason with its step. Info shows the same list.
  • Actions. Open goes to /new-run?saved=<id>, and the builder loads that recipe at Review, as the Saved recipes dialog does. Info shows the recipe step by step and its JSON. Duplicate saves a copy as " (copy)", or the next free number. Export downloads the JSON. Delete asks first, in the same dialog style as Workspaces. Import, on the title line, sends a file to the strict import. A toast shows the reason for a refusal. The writes use the API of #PR_C.
  • Remote. A browser on another machine sees Import, Duplicate and Delete greyed (aria-disabled), with the reason as the tooltip. Open, Info and Export work.
  • Empty. With no saved recipe, the page shows "No saved recipe yet. Build one →", a link to the run builder, and the Import button. A browser on another machine gets no link, because it cannot save. A notice right under a page title now has the gap that a table's controls row has (style.css).
  • Refactor. new_run and the page get the builder data from one helper (_builder_data).
  • New icons in the sprite: i-recipe (the page), i-copy (Duplicate) and i-export (Export). Open uses i-newrun, Info uses i-info as Settings does, and Import uses i-import.

Touches

routes (app/web/builder.py) · templates and static (app/templates/builder/recipes.html, app/templates/base.html, app/static/builder/recipes.js, recipes.css, command-builder.js, command-builder-core.js, style.css) · docs (docs/builder.md)

Before and after

Before: no Recipes page. The Saved recipes dialog in the builder is the only view.

After:

Light Dark
Table, 1280 px table-1280-light table-1280-dark
Table, 390 px table-390-light table-390-dark
Info, 1280 px info-1280-light info-1280-dark
Info, 390 px info-390-light-many-agents info-390-dark
Delete confirmation delete-ask-1280-light delete-ask-390-dark

Other:

search-1280-light duplicate-1280-light import-bad-1280-light open-builder-1280-light remote-1280 remote-actions-1280 remote-390

Verified

On a fresh clone of this branch, in a python:3.12 container with Node and uv:

  • python3 app/tests/parallel.py: 2249 tests in 127 modules, OK.
  • python3 app/tests/parallel.py app/ui_tests: 611 tests in 51 modules, OK. New: test_recipes_page.py (4 tests: the sidebar link, the columns and a row for each recipe, the empty state; Import, Duplicate and Delete greyed for a non-local client while Open, Info and Export stay; /new-run?saved=<id> loads the recipe, and an unknown id gives 404; under Node, the page script over the page data names a missing agent, a missing standard benchmark and a runner that is not offered, and finds free copy names).

Browser check in Chromium (Playwright 1.49.1) with six saved recipes: two Harbor recipes, one with ten agents, one Harbor recipe with an agent that is not offered, two standard recipes, and one standard recipe with a benchmark that is not in the catalogue.

  • At 1280 px and 390 px, light and dark: the table, the warning on the two bad rows, the cut agent list with its full tooltip, Info and the Delete confirmation.
  • Search, Export (the file is the recipe), Duplicate, Import of a good file, Import of a file with one bad recipe (toast: "Not imported: recipe 2 of 2: …"), Import of a file that is not JSON, Delete after the confirmation, and Open into the builder at Review.
  • With X-Forwarded-For set: Import, Duplicate and Delete greyed with the reason. Info works.
  • No page JavaScript errors. The only console error is the 400 answer to the bad import, which the check asks for.

Risk

The page reads the recipes and writes only through the API of #203, so a fault here cannot change a recipe that you do not act on. A check that is wrong shows a wrong warning, and the builder still checks the recipe again at Review and at Run. A host that Settings no longer has is not in the warning, because the builder checks hosts only at Review and Run. A revert removes the page and ?saved=. The saved recipes stay as they are.

… one place each

The Recipes page reads the same data the run builder does, and checks a
standard recipe with the same compile as the builder's Review step.
A table of the saved recipes, a column per builder step, with search,
Columns, Sort by and the pager. Each row opens its recipe in the run
builder (/new-run?saved=<id>), shows it step by step with its JSON,
duplicates it under a free name, exports it or deletes it after asking.
Import takes a recipe file or a list through the strict import.

A row whose recipe the builder's own check refuses shows "Cannot run
here", its reasons in the tooltip and in Info: the recipe core's check
for a Harbor recipe, the Review step's compile for a standard one.

A browser on another machine gets Import, Duplicate and Delete greyed,
the reason as the tooltip.
@harenome
harenome merged commit a5b1601 into ThinkFlowLab:main Nov 18, 2026
2 checks passed
@harenome
harenome deleted the feat/recipes-page branch November 18, 2026 11:17
aitianshi pushed a commit that referenced this pull request Nov 19, 2026
…viders, set its own limits and prove a merged null control (#133)

## What

A benchmark of the standard can now say more about how its units run and
end, which a computer-use benchmark like OSWorld2 (#134) needs. `gated`,
the new limits and `prove`'s new options are opt-in. The relay's rules
and the rules of how a unit ends change for every benchmark of the
standard; Risk lists them.

- **Gated tasks** (`gated: true`): a unit's failure that the relay's
records explain, a model endpoint's (the agent's, the judge's or the
simulated user's), says in its `result.json` the status and class alone,
never the provider's words, which can quote the request. The trial's
call records keep the provider's words, and an error a driver raises
itself is kept as the driver words it.
- **A benchmark option's values by name** (`names`: {value: name}): the
builder's cards and select, Review and the benchmark's page show the
name, `about` stays its tooltip, and a recipe and the CLI keep the
value. OSWorld1's agents are named as their upstream spells them.
- **An option on its own step** (`step`): `where`, where the work runs,
on the Where step, one value as fixed, and in Review's Where row;
`dataset`, what some datasets need, on the Dataset step for those
datasets. OSWorld1's VM provider is on Where.
- **Limits of a benchmark's own:**
- `limits.score_seconds`: how long `score` may take, the benchmark's
level below `--timeout score=` (#191); the Limits step shows it as the
benchmark's;
- `limits.call_seconds`: how long the relay waits for one model call,
also `relay serve --timeout`;
- **How a unit ends:**
- a driver's error that begins `EnvironmentSetupError: `,
`VerifierError: `, `CancelledError: ` or `CauseUnknownError: ` keeps
that kind;
- the agent's own last call refused as longer than the model's context
window is `ContextWindowExceededError`, the agent's;
- a last 429 is read from the agent's own calls in the order they began,
so a side model's later 200 does not cover it;
- a unit a guard's stop ended keeps the score its driver gave, its own
last call's 429 notwithstanding, unless that 429 said the credit is
spent. `score` can say the stop's reason (`verifier_result.stopped`);
- with #154's `--retries`, a unit a guard asked to stop and a full
context window are not tried again;
- for an upstream's own answerer: `OutputLengthExceededError` only when
its last reply has neither text nor a tool call. When the answerer fails
and its last reply was a Responses object answered 200 and `failed`,
that is an API error.
- **The relay:**
- a non-streamed 200 to a model call with no reply in it reaches the
caller and the record as a 502, or the 429 or 402 of an error it held
(the record keeps `answered: 200`). #124 leaves counting an error inside
a 200 to a pull request of its own; this one takes only a 200 with no
reply in it;
- a stream cut off midway, after the caller had its 200, is recorded as
a 502, and its booking against the spending cap stays whole;
- a call refused with 400, 413 or 422 records each inline image it sent
by its declared type, the type its bytes show and its size, never a
byte;
- a `stop` file beside the relay's records: the relay notes in it an
agent's call refused for the model's context window, and that note
refuses nothing. Once a benchmark's guard writes a line of its own
there, the agent's later calls are refused unsent, while side models and
the judge's and simulated user's routes go on;
- a failed Responses object with no usage records a cost of 0, and its
booking is kept.
- **A job's name:**
- a run holds its job name until its process ends, and one under a name
another run still holds is refused before it writes anything;
- a rerun into the same job moves each rerun trial's earlier records
(`.work/relay`, `result.json`, `verifier/`, `agent/`) aside to
`.work/earlier.<ns>`.
- **prove and merge:**
- `nullius standard prove ... --bench-option NAME=VALUE` sets the
benchmark's own options for the units check and the controls;
- `--job nop=MERGED` (or `oracle=`) checks a merged launch of a control
against its bound instead of running it, and every trial's own record;
- `merge` accepts shards that differ in benchmark options and keeps each
shard's in `merged_from`;
- `merge` refuses, unless `--force`, a trial whose record names another
job, benchmark, version, control or agent, and a job itself merged
despite differences. It lists the shards absent from a merge only when
the launch has at most 1,000.
- **Smaller:**
  - a trace that cannot be built leaves `agent/trace.error.txt`;
  - an Original's trajectory keeps its tool calls;
  - each call's record keeps the request's sha256;
  - `thinking_tokens` count as reasoning;
- a setup's `comparable` can list the models whose runs compare with the
paper, and the builder's message for another model says so;
- a test checks that every file a host's snapshot takes of a bundled
entry is package data.

**Read first:** `failure`, then `_hollow` and the stop in
`Relay._forward` (`app/nullius/standard/relay.py`). Next, the outcome
rules in `Run` (`app/nullius/standard/run.py`), then `names` and `step`
in `recipe.compile` and `benchOptions` in the builder, the `--job` check
in `prove.py` and the trial check in `merge.py`. Then
`test_standard_gated.py`, `test_standard_context_window.py` and
`test_standard_controls.py`. The rest is the definitions' checks, the
CLI options, the sidecar's argument and the pages.

## Touches

the standard (`app/nullius/standard/`: `relay`, `run`, `prove`,
`definitions`, `merge`, `job`, `cli`, `container`, `recipe`, `fidelity`,
`retries`) · the builder (`app/static/builder/`, `app/web/builder.py`:
the score's default on the Limits step, an option's names, and an option
on its own step) · the benchmark page (an option's names) · OSWorld1's
definition (its agents' names, its VM provider on Where) · docs and site
(`docs/standard.md`, `docs/cli/run.md`, `docs/cli/read-jobs.md`,
`docs/builder.md`, `docs/settings/model-providers.md`)

## Verified

- At `e768c632`, on main at `db6e8f96`, on Python 3.12: backend suite
(`python3 app/tests/parallel.py`), 2310 tests in 129 modules, OK; route
suite with Node on PATH, 639 tests in 53 modules, OK; the wheel's check
(`app/packaging/verify.py`), OK.
- **An option's names and its step, in a headless Chromium, light and
dark, 1400 and 390 px,** with #134 on top: OSWorld1's agents by name in
their select, OSWorld2's in their cards; the VM provider of each as a
fixed value on the Where step, shown only while a participant that takes
it is chosen; OSWorld2's Website host suffix on the Dataset step for All
tasks, not for Offline, no services; Served by empty on any choice of
model. No page error.
- **The 12 changed test modules on main's source** (`5bcfd82d`): 9 fail,
2 pass, and 1 does not end.
- `test_standard_context_window`: 19 of 20; `test_standard_controls`: 17
of 43; `test_standard_relay`: 8 of 58; `test_standard_gated`: 4 of 6.
- `test_standard_run`: 2 of 50; `test_standard_trace`: 2 of 20;
`test_standard_parts`, `test_standard_served_by` and
`test_standard_units`: 1 each.
- `test_standard_shard` does not end within 4 minutes: one of its merges
names 2^64 shards, and main's `merge` lists every absent one.
  - `test_standard_versions` and `test_dependency_surface` pass on both.
- **Covered in #134, not here:** that `limits.call_seconds` reaches the
relay and the sidecar's `--timeout`. OSWorld2 is the first entry to set
it (`test_osworld2_muse.py`).
- **#124's own tests.** Three were changed for two deliberate
differences: a 200 with no reply is a 502, and a failed Responses object
is read as failed. The rest of `test_standard_served_by` passes as main
has it; `write_shard` in `test_standard_shard` takes #137's `reward`
beside this branch's `bench_options`.
- `PYTHONPATH=app python3 -m nullius standard check`: ok. `python3
app/tests/catalogue_docs.py --write`, then the check: up to date.
`python3 app/tests/field_contract.py`: ok. `app/site/readme_media.py
--check`: current. The docs check reports the same two findings in
`AGENTS.md` as on main.
- Not run: a benchmark already on main, live, on this branch; the
container backend with `--timeout`; the docker tier; the site build.

## Risk

- **For every benchmark of the standard:**
- a non-streamed 200 with no reply is a 502, and a stream cut off midway
is recorded as one;
- the agent's own last call refused for its context window is the
agent's error (`ContextWindowExceededError`), where it was an API error
outside its rate;
- an agent's last 429 followed by a side model's 200 makes the unit
`rate_limited`, where its score stood;
- an upstream answerer whose last reply was a `failed` Responses object
is an API error, where it was the agent's;
- a driver's error of a kind it names: a setup error or a stop is left
out of the agent's rate, and a verifier error or one of no known cause
counts as unknown. A wrong kind would move a unit between them;
- `prove` runs no control when the units check failed, and fails a
control job with a unit that has neither a score nor an error;
- a run under a job name another run still holds is refused, and a rerun
into an existing job moves the rerun trials' records aside;
- `merge` refuses a trial whose record names another job unless
`--force`.
- **Records.** Stored records gain fields and keep the old ones.
- These commits sit on main at `db6e8f96`, with #124, #150, #191, #154,
#174 and the Recipes page (#205). #124's `--served-by` code is main's as
merged; #191's time limits come first, so `--timeout score=` is used
over `limits.score_seconds`, and a unit's cap on model calls
(`UnitCallCapError`) is read before this PR's outcome rules. #154's
`--retries` replaces the one more run of a rate-limited unit, so the
earlier `limits.run_again: false` that switched it off is gone; main
counts a provider's 402 as a usage limit, and the tests follow. The
first and fourth commits gave a definition pins of its serving
providers, defaults of Served by after review; the last commit removes
them, as the reviewer asked, since no benchmark declares one: the diff
over main has none, and Served by is the user's alone, as in #124.
- The fifth commit shortens this PR's docs and messages. No logic
changes.
- **Names and Where.** An option's names change only what is shown; a
recipe saved before keeps working. An option of where the work runs
leaves the Agent step for Where, and one some datasets need is asked on
Dataset: OSWorld1's VM provider is now on Where.
- Undone by reverting the commits.

---------

Co-authored-by: Oleg <247799477+sirotaoleg@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a Recipes page to manage saved builder recipes

1 participant