Repository navigation
Closed
Feature
Copy link
Labels
P2Planned: limited in scope, has a way around it, or a missing featurePlanned: limited in scope, has a way around it, or a missing featureenhancementNew feature or requestNew feature or request
Description
Priority
medium. The change is in the web layer. It changes the Compare routes and templates, adds a JSON route for a workspace's runs, and changes the table toolkit that other tables share. In the backend, one parser accepts an empty column.
Components
- app/nullius (backend)
- app/web (routes)
- templates and static files
- docs and the site
- catalogue (benchmarks, agents, capabilities)
- the standard (ports, kits, relay, runs)
- CI, workflows and skills
Today
- Compare needs two runs. With fewer, it shows a separate picker page (
/compare/pick) with two steps: a Left and a Right workspace, then a Left and a Right run. To add a third run, you go back to All runs and tick it there. - Each column header shows the agent, the job id and the workspace name with no label. The table has no Benchmark row.
- A sharded run is a column per shard. A Runs page
job=link of one shard compares that shard only. - The cohort switch sits above the page with a box of explanation under it. A "Side by side" heading, a row of "Order columns:" buttons and a "Task by task" heading with an explanation line follow. Both tables are on one page, one under the other.
- No number shows its difference from another column.
- The table header does not stay in view when the page scrolls. A long table loses the names of its columns.
- In the table toolkit's controls row, buttons, text buttons and filter pills have different heights.
Wanted
- Compare opens with two columns. With one run or none, an empty column waits beside the run.
- Workspace and Run are the first two rows of the table, labelled in the left column as the metrics are. Each column picks a workspace, then a run, from a searchable dropdown. A list opens only on its button, or after a workspace change.
- A run that another column holds is greyed in the lists. A sharded run is one column, with a link to each shard.
- A Benchmark row above Dataset.
- The first column is the baseline. Each summary number of the other columns shows its difference from the baseline, in the row's unit.
- Each column has Move left, Move right and Remove. Add a run adds an empty column.
- The table toolkit's controls row heads the table. Its views are the cohort toggle, a Side by side / Task by task toggle and an Order menu. Its action is Add a run. One view shows at a time. Every link keeps the view and the cohort.
- Task by task has the same column controls.
- The explanation box, the "Side by side" heading and the Task by task explanation line go. The warnings that ask for an action stay: no shared cohort, and datasets that differ.
- In the table toolkit, rows with
data-stickystay in view under the controls row while the page scrolls, until the table ends. Every control in the controls row has one height.
Acceptance
-
/comparewith no run or one run shows two columns. An empty column of two has no Remove. - Workspace and Run are rows of the table, one cell for each column.
- A column's run list is searchable, newest launch first, the baseline's dataset first, checks marked. It does not open on load or after Add a run.
- A workspace picked lists its runs in place (
/compare/runs). A run picked opens the comparison with that run in its column. - A run that another column holds is greyed in a list, with a tooltip that names the column. A click or Enter on it does nothing.
- A sharded run is one column with its trials from every shard and a link to each shard. A shard's id on a
pick=orjob=link stands for its whole run. - The table has a Benchmark row above Dataset, with a dash when the catalogue does not know the dataset.
- Each summary number of columns 2 to N shows its difference from the first column, in the row's unit. The row's direction sets its colour. A check has no colour.
- Move left, Move right, Remove and Add a run are links that keep the cohort and the view.
- The controls row holds the cohort toggle, the view toggle, an Order menu with the
i-reordericon, and Add a run. The Order menu holds Group by agent and Alternate agents. -
?view=tasksshows only the task table, with the same column controls. Side by side has no parameter. - The explanation box, the "Side by side" heading and the Task by task explanation line are gone. The no-shared-cohort warning and the datasets warning show in both views.
-
/compare/pickredirects to/comparewith its picks. - A table's
<tr data-sticky>rows stay under its controls row while the page scrolls, until the table ends.data-sticky="wide"rows do this only from 900 px wide. - The controls in a table's controls row have one height (
--tool-h). - The field guide and the results guide describe the columns, the baseline and its difference.
Dependencies and related
- feat(web): build each table from one column list, with search, sticky columns and aligned numbers #199, the table toolkit (merged). Compare uses its controls row, its search matching and its sticky columns.
- feat(web): build the catalogue cards from one field list, with search and filter pills or dropdowns #201, the card toolkit. It shares the controls row.
- feat(runs): read a task's repeated attempts together, by pass rate, with pass@k and pass^k #146 and Compare flags a task as disagreeing from its last attempt only, and shows no pass@k or pass^k #130, repeated attempts and pass@k. The pass@k rows keep their place in the side table.
Metadata
Metadata
Assignees
Labels
P2Planned: limited in scope, has a way around it, or a missing featurePlanned: limited in scope, has a way around it, or a missing featureenhancementNew feature or requestNew feature or request
Activity
feat(compare): compare runs in columns, each a workspace and a run, u…
feat(compare): add a Chart view of the summary rows (#218)