Repository navigation
Labels
P3When convenient: wording, layout, testsWhen convenient: wording, layout, testsenhancementNew feature or requestNew feature or request
Description
Priority
low. The change is in the web layer. It adds a view to Compare and a small chart module in the backend. It changes no stored data and no other page.
Components
- app/nullius (backend)
- app/web (routes)
- templates and static files
- docs and the site
- catalogue (benchmarks, agents, capabilities)
- the standard (ports, kits, relay, runs)
- CI, workflows and skills
Today
- Compare has two views, Side by side and Task by task. Both are tables.
- To see how far apart the runs are on a metric, you read the numbers row by row. With seven runs, the size of each difference is hard to see.
- The only charts in the app are the trace distributions on a trial page. They put span durations in bins on a log axis, and they do not fit one value per run.
Wanted
- A third view, Chart (
?view=chart), with the same columns, cohort and controls row. - Each chart is a card. Its header is one row of toolkit buttons: Style (Bars, Dots, Scatter), then Data for Bars and Dots, or X and Y for Scatter, and Remove at the end. Each metric button shows the name of its metric and opens a menu of the summary rows.
- Add chart, in the controls row beside Add a run, adds a card.
- Bars and Dots draw one mark for each run on the metric's own axis and unit, with the baseline marked. Scatter draws one dot for each run, with a number beside it and a key under the plot.
- The URL holds the charts, one
chart=value for each card, in order:chart=bars:adjusted_rate&chart=scatter:cost,adjusted_rate. Every link keeps them. - The values are the cells of the side table. A missing value has no mark. A check (
nop,oracle) is drawn apart and never ranked. - A new style is one drawing function and one entry in a registry of styles.
Acceptance
- Without
chart=,?view=chartshows three cards: Bars of the headline rate, Bars of the cost, and a Scatter of the cost (X) against the rate (Y). The headline rate is the adjusted rate, or the mean score for a benchmark that only scores. -
chart=with no value shows one Add chart button and no card. - A card's Style menu changes only that card. A change keeps the card's first metric, and a new Y is the headline rate, or the cost when the rate is X.
- A card's Data, X or Y menu changes only that axis of that card. Each button shows its metric's name.
- At 1280 px a card's header is one row. Where a row is too narrow, whole buttons move to the next row, and no name breaks over two lines.
- A card's Remove (the × of a column's Remove) removes only that card and keeps every other parameter.
- Add chart appends a Bars card of the headline rate.
- Every link on the page keeps the view, the charts and the cohort.
- Each value in a chart is the text of the same cell in the side table. A missing value shows a dash and no mark.
- Scatter shows a dot for every run that has both values, also when several runs share one point. Each dot has a number, and no number goes outside the plot.
- A check is hollow and named "(check)". No mark is shown as best.
- Each run has one colour in every chart, from its column's place: eight
--series-*tokens, for every theme, cycling after eight. The palette passes the data-viz validator (contrast 3:1 on the card, adjacent ΔE 8 for colour-vision deficiencies) and reads as no outcome colour. A check's outline has its run's colour, and the Scatter key shows a swatch beside each name. - Each chart is an SVG with
role="img"and a title, and a table for a screen reader gives every value as text. - At 390 px the cards stack, and the page does not scroll sideways.
- The trace distributions on a trial page do not change.
Dependencies and related
- Compare runs in columns, each column a workspace and a run, with the table toolkit's controls #206 and feat(compare): compare runs in columns, each a workspace and a run, under the table toolkit's controls #207, the Compare redesign. The Chart view uses its columns and its controls row.
- feat(runs): read a task's repeated attempts together, by pass rate, with pass@k and pass^k #146, repeated attempts and pass@k. A later Lines style can draw pass@k over k.
Metadata
Metadata
Assignees
Labels
P3When convenient: wording, layout, testsWhen convenient: wording, layout, testsenhancementNew feature or requestNew feature or request
Activity