Skip to content

feat(builder): draw the Benchmark step with the Benchmarks page's cards, as cards or a table - #267

Merged
harenome merged 8 commits into
ThinkFlowLab:mainfrom
harenome:feat/builder-benchmark-cards
Nov 20, 2026
Merged

harenome merged 8 commits into
ThinkFlowLab:mainfrom
harenome:feat/builder-benchmark-cards

Conversation

@harenome

@harenome harenome commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #266

What

The builder's Benchmark step now shows the same listing as the Benchmarks page, drawn with the shared card and table toolkits. Before, command-builder.js built its tiles from three sources: the standard payload, the Harbor list from launch.py, and catalogue lookups.

  1. One listing. recipe.benchmarks(cat) gives the listed benchmarks with their datasets, tasks, machine needs, what they evaluate (agents, or a kind of capability) and each other version's tasks. catalogue.benchmark_cards makes the cards from it for the Benchmarks page and the step. The Benchmarks page HTML does not change, except for one new sprite icon.
  2. A fragment route. GET /new-run/benchmarks returns the step's grid (builder/benchmarks.html), with ?view=table for the table. The builder loads it once per view, with the same loader as the Agent step's grid.
  3. Single choice. A pick with one: true is radio-like. A click on a card chooses it and lets the others go. A second click keeps it chosen. In the table, the narrow left column has a radio, and a row click chooses it. Toggle picks do not change.
  4. Cards. Mark, name, and a check when chosen. An "Agents" or "Capabilities" badge. A Tasks stat. The category tag, and "Community" for a community benchmark. For a benchmark with versions, the Tasks stat shows the version that the recipe names, and it changes in place when the version changes (OSWorld1: 368 for its default, 369 for main).
  5. Controls. Search, the category pills with All, an Evaluates dropdown (Agents, Capabilities) with a new i-target icon, a Filter menu (Runner), Sort by (Name, Tasks), and the cards/table toggle. The toggle loads the other view in place: the page does not reload, the choice and the draft stay, and localStorage keeps the viewer's choice.
  6. Chosen stays visible. The chosen benchmark passes any search or filter, also after a recipe import.
  7. Toolkit fixes for every page. A card table's name wraps only between words, and the table scrolls sideways instead. Inside a chosen pick card, a tag or a mark filled with --accent-soft and a tag outlined with --border take a shade nearer the accent, so they keep their fill and outline in both themes.
  8. Field contract. Benchmark.versions is pinned. feat(catalogue): add OSWorld desktop-env-v1.0.3 as OSWorld1's default version #261 added the field without a pin refresh.

Touches

backend (nullius/standard/recipe.py) · routes (web/builder.py, web/catalogue.py) · templates and static (_cards.html, base.html, builder/benchmarks.html, tables.js, catalogue/catalogue.css, builder/command-builder.js, command-builder.css) · docs (builder.md) · tests (field_contract.json)

Before and after

Before After
01-before-benchmark-step 01-after-benchmark-step
02-before-osworld1-chosen 02-after-osworld1-chosen
03-before-evaluates-chips 03-after-evaluates-menu
04-before-benchmark-step 04-after-table-view
05-before-benchmarks-page 05-after-benchmarks-page
06-before-benchmark-step-390 06-after-benchmark-step-390
07-before-benchmark-step-dark 07-after-benchmark-step-dark
  1. Builder, Benchmark step as it opens (GAIA is the default). 2. The same, with OSWorld1 chosen. 3. The Evaluates menu open (before: the Any, Agents, Capabilities pills). 4. The table view with OSWorld1 chosen (before: the old step). 5. Benchmarks page, unchanged. 6. Benchmark step with OSWorld1 chosen, 390 px wide. 7. The same, dark theme.

Verified

  • In a rootless Podman container (python:3.12, Node 20): python3 app/tests/parallel.py: 3108 tests in 159 modules, OK.
  • pip install -r app/requirements.txt, then python3 app/tests/parallel.py app/ui_tests: 703 tests in 58 modules, OK.
  • python3 app/tests/catalogue_docs.py --write and python3 app/tests/field_contract.py --write leave the tree unchanged.
  • app/ui_tests/test_recipe_execution_fixtures.py and its fixtures are unchanged.
  • New tests:
    • test_benchmark_listing.py: The listing is the catalogue's listed benchmarks with their family. A benchmark with versions counts each version's tasks.
    • test_builder_benchmarks.py: The step lists the Benchmarks page's benchmarks, each a single choice. A benchmark with versions carries each version's tasks. The controls are in place. The table chooses one with radios. The Benchmarks page stays a page of links.
    • test_cards.py: A single choice is marked on its card. It lets the others go and stays chosen. A row click chooses its radio.
  • In a browser, on local servers with empty profiles: the seven views above on main and on this branch. No page error. The /benchmarks HTML of both servers differs only in the i-target symbol.
  • In the same browser: switching to the table kept OSWorld1 chosen. Its card showed 368 tasks, then 369 after a change to main. A search for "gsm" left GSM8K and the chosen OSWorld1. A click on GSM8K left only GSM8K chosen.

Not checked: a recipe import in the browser, and a real run from the step.

Risk

  • Every card table's name now wraps only between words. A long name widens its column, and the table scrolls sideways.
  • Every chosen pick card, the Agent step's too, changes the shade of its tags and marks. Unchosen cards do not change.
  • The step needs one more request per view. If it fails, the step shows one line, and the next render asks again.
  • To undo, revert the commits. No stored data or recipe format changes.

recipe.benchmarks gives the catalogue's listed benchmarks with their
datasets, tasks, machine needs and what they are run with (agents, or a
kind of capability). The Benchmarks page draws its cards from it through
catalogue.benchmark_cards, with the same output as before, so the
builder's Benchmark step can draw the same cards.
A pick with one: true is a single choice: choosing its card lets the
others go and choosing it again keeps it; in a table its box is a radio.
Toggle picks are unchanged.
…ds, as cards or a table

GET /new-run/benchmarks returns the step's grid: the Benchmarks page's
cards as single-choice pick cards, or a table with ?view=table. The
category pills stay; Run with (agents or capabilities) is a menu of its
own beside Filter (Runner), Sort by and the cards/table toggle. The
builder loads it once per view, switches the view in place and keeps it
per viewer, and presses the recipe's benchmark, which stays in view
under any filter, also after a recipe import. The step's own tiles,
search and chips are gone.
A name column that takes the spare width broke a long name inside a
word on a phone (LongMemEval as "LongMemEv al"). The name now wraps
only between words, and the table scrolls sideways instead.
The benchmarks' listing carries each other version's tasks (OSWorld1's
main pin: 369, its default tag: 368). The Benchmark step's card and row
of a benchmark with versions hold them as data-tasks, and the builder
shows the chosen benchmark's count for the version its recipe names, in
place when the version changes, as the step's tiles did. The card
toolkit takes data attributes for a card and its row.
…Evaluates, with a target icon

"Run with" read oddly beside the runner, and the plug did not say it.
The menu, its column and its field are now Evaluates (Agents,
Capabilities), led by a new i-target icon in the sprite: three
concentric circles in the sprite's line style. i-plug stays for
Settings.
A chosen pick card takes --accent-soft as its background, so what is
filled with it inside the card (a benchmark's category tag, an agent's
initials) lost its fill, and in the dark theme an outlined tag (an
agent's protocol) lost its outline. Inside a chosen card both are now a
shade nearer the accent, so they stand out as on any card, on every pick
card, in both themes. Unchosen cards do not change.
ThinkFlowLab#261 added the versions field to the catalogue's Benchmark without
refreshing the pin; this records it, so its removal fails the check.
@harenome
harenome merged commit 71d0ff5 into ThinkFlowLab:main Nov 20, 2026
2 checks passed
@harenome
harenome deleted the feat/builder-benchmark-cards branch November 20, 2026 13:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Builder: the Benchmark step draws its own tiles, apart from the Benchmarks page's cards

1 participant