Phase 1B harness: measure what add and remove did, and keep pacing out of latency by ryandmonk · Pull Request #99 · aestheticfunction/dspack-studio

ryandmonk · 2026-10-12T10:53:10Z

One narrowly scoped, no-paid-call measurement-correctness change to the Phase 1B harness, made before stage 3 (Astryx first) is authorized. It touches the harness and reporting layer only: packages/block-bench/live/ and one section of docs/BLOCKS.md.

What changes

Add and remove refinements are checked for the operation they ask for (live/operation.ts, new).

The problem it closes:

A passing refine-add or refine-remove cell only proved that the surface passed the gates.
changed meant any change, so a refinement that rewrote a title, or did the opposite, still counted as a pass.
The scripted dry run shows this: every add and remove passes, and none changes a list.

How the check works:

It compares the refined surface with the prior one the same arm built, list by list: a node's children, and the arrays in its props.
Each list is keyed by the node that owns it:
- A node in a list whose id occurs exactly once in both surfaces is matched by component and id in whichever list it sits. So an item put first, a table moved into a new container, or reordered siblings leave its lists alone.
- Any other node in a list (no id, a renamed id, an id on one side only, a repeated id) is placed by its rank among same-component siblings not matched by id. A renamed or repeated id therefore never reads as lists removed and added again, but such a node that moves can read as "mixed".
- On the smoke surfaces, nearly all Astryx nodes carry ids; shadcn nodes mostly do not.
An add must lengthen a list and shorten none; a remove, the reverse. Each list is judged on its own.
It is recorded as operation on the cell.
It is counted per condition and per block. Misses that left the surface unchanged are counted apart, because an unchanged surface is the right answer where a template does not apply to the surface.

It never enters the pass rate. ADR-B3's pass is frozen with the corpus and its scoring. Instead, each decision gets a table of sensitivity readings:

Each reading recomputes the decision with some of those refinements counted as failures. There is one for each way of counting them that applies:
- the missed ones;
- the missed and mixed ones;
- where some miss left its surface unchanged, both of these again with only the misses that changed their surface.
Each row gives the recomputed verdict, its clarity, the pass rates with counts, and the token reduction.
A conclusion drawn from every reading says whether the verdict, or only its clarity, changes in any of them.
An unchanged surface is ambiguous. It is the right answer where a template does not apply, but it is also what an ignored instruction leaves: a block refinement that re-sends its prior call expands to exactly the same surface. So the readings that leave unchanged misses out can excuse real ones, and the report says so beside the readings and under the by-block table.
The verdict itself is unchanged.

Latency excludes the harness's own pacing.

The pacer measures its spacing and backoff waits (waitedMs) on the monotonic clock a cell's latency uses.
Each cell keeps timings.pacingMs and an execution set of figures beside the unchanged wall-clock figures.
Latency medians read the execution set. p50 emit and p50 pacing are printed.

Below the pre-registered minimum, nothing is called clear. The minimum is 20 prompts behind a criterion: the prompts every decision arm evaluated, and for the token figure, those of them metered in both the block and production arms (meteredCohort).

Below it, every criterion with an interval is insufficient, never "clearly met". This follows the addendum: indicative only below about 20 prompts.
The decision is insufficient when the prompts in common, or an arm's threads, fall short of the minimum.
A token figure on too few metered prompts leaves a pass uncertain, while a fail clearly missed on a pass rate stays clear, as the addendum defines a clear decision.
A criterion with no interval at all stays "not measurable".

Reporting:

Paths table: paths, fallbacks by classification, budget left at fallback, invariant failures and budget exhaustion, for every arm and condition, plus every condition pooled.
Labels: the Input and Output columns are labelled per success.
Parity: a checklist with nothing to check is counted apart, not scored 1.0. The report notes that the frozen checklist holds no action names, though the plan behind ADR-B3 lists them.

The frozen boundary

These are all unchanged:

the corpus, labels, structural checklists, conditions, thresholds and scoring;
request bytes;
block selection, expansion, rendering, call budgets and application behavior;
the named and positional implementations.

git diff --stat 03c9465..HEAD is empty against all of these: live/corpora, results.json, RESULTS.md, goldens, src, apps, composer-core, contracts, live/providers and docs/blocks (the addendum).

Check	Value
Corpus digest	`180f4480…4231`, matching its pin
Harness code identity	Changes, as any harness change does. Stage 3 will pin the merged value.

Evidence

Fail-first, one commit of tests before each implementation:

Tests	Failing on	Fixed by
`07c8a6c`	the base: 13 fail, plus one test file that cannot load	`e044e03`
`7b50ed7`	`e044e03`: 14 fail	`3abca36`
`48aaae3`	`3abca36`: 9 fail	`b0fb8e5`
`aaa01b1`	`b0fb8e5`: 12 fail	`d71f2cd`
`1e6177f`	`d71f2cd`: 3 fail	`e057c44`

Block-bench: tsc is clean, and 233 of 233 tests pass in 16 files, including:

src/bench.test.ts: the committed request goldens;
live/corpus.test.ts: the frozen pin;
the dry run.

Root:

pnpm typecheck is clean.
pnpm test on e044e03: everything passes except one apps/agent test this PR does not touch (src/project.test.ts > restoredConflict). It times out at 5 s in the review container and does the same on main 03c9465 there. CI ran it green.

Request hashes are unchanged:

apps/composer/app/hosted-build.test.ts: 43 tests, including "the real call sites send the Phase 0 request bytes".
lib/blocks-ratchet.test.mjs passes.
results.json is unchanged.

The check on real surfaces. Every pure single-node removal and insertion was simulated on the 35 smoke surfaces:

Surfaces	Removals read as "remove"	Insertions read as "add"	The rest
Astryx	392 of 392	2,512 of 2,512	none
shadcn	653 of 788	2,797 of 3,640	"mixed", because id-less nodes are matched by place

None reads as "none" or as the opposite direction. A false "mixed" moves a cell only into the readings that count mixed changes, never into the verdict.

Independent reviews of e044e03, 3abca36, b0fb8e5 and d71f2cd confirmed:

the frozen boundary;
request-byte safety;
identical records from 3,600-cell scripted dry runs at base and head;
decide() outputs identical to base apart from the additive meteredCohort and the clarity labels where a minimum is not met, over 2,300 random record sets; the readings and their conclusion matched an independent recomputation in 3,307 more.

Their findings, and Copilot's, are fixed in the commits above.

The final independent review of the exact head e057c44 found no unresolved material concern. It compared syntax trees to confirm that d71f2cd..e057c44 changes only strings, comments, docs and tests.

Not in this PR

Making the operation check binding.
Action names in parity. The frozen corpus has no per-prompt expected actions, and emitted action names are slugs of node ids, which differ by arm.
Fixing the shared refinement templates:
- records-table "add" names ticket fields;
- auth-form "remove the provider buttons" cannot apply to a form without providers.
Matching id-less shadcn nodes by sequence alignment rather than by place.

The first three are frozen-input changes and are proposed to the owner as a versioned amendment, not made here.

🤖 Generated with Claude Code

https://claude.ai/code/session_01R6Ax8qvfN3gYUC85oSGnoi

Tests only; each fails on the unchanged harness (13 assertions, plus the new operation.test.ts, which cannot load its module yet): - add and remove refinements: an add must lengthen a list in the surface and shorten none, a remove the reverse, judged from the prior and refined surfaces alone (operation.test.ts, arms.test.ts). The scripted twin passes every add and remove by rewriting a text node, and the dry run must say so (dry-run.test.ts). - the check sits beside the rule: the verdict is unchanged, and a sensitivity recomputation says when the verdict rests on refinements that did not do what they asked (methodology.test.ts). - latency is execution time: the pacer measures its own waits, and a cell keeps them apart from every latency figure (pacing.test.ts, hosted.test.ts, metrics.test.ts). - no criterion is called clear below the pre-registered minimum: it is insufficient, and the report never prints "clearly met" there (methodology.test.ts; the cohort-10 case now expects insufficient). - parity leaves out checklists with nothing to check; the report labels input and output per success, prints emit and pacing, tabulates paths, fallbacks and invariant failures for every arm and condition, and states that action names are not in the frozen checklist. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R6Ax8qvfN3gYUC85oSGnoi

…t of latency Harness and reporting only. No request byte, block selection, expansion, rendering, call budget or application behavior changes, and no frozen input (corpus, labels, checklists, conditions, thresholds, scoring) changes. - live/operation.ts: a passing add or remove refinement is checked against the prior surface the same arm built, by the length of every list (children and arrays in props), so an add must lengthen a list and shorten none and a remove the reverse. Recorded on the cell (`operation`), counted per condition, never in the pass rate, which ADR-B3 froze. - metrics.ts: each decision gets a sensitivity reading, recomputed with the refinements that missed their operation failing, and again with mixed changes failing too, and says whether the verdict rests on them. The verdict is unchanged. - pacing.ts and arms.ts: the pacer measures its own spacing and backoff waits; each cell keeps `pacingMs` and an `execution` set of figures beside the wall clock, and latency medians read the execution set. - metrics.ts: below the minimum of 20 prompts in common, every criterion and the decision are insufficient, never clear (the addendum's ADD-19). Parity leaves checklists with nothing to check out of the mean. - report.ts: per-success labels for input and output, p50 emit and p50 pacing, paths, fallbacks and invariant failures for every arm and condition with the pooled rows, the add and remove column and line, and a note that the frozen checklist has no action names. - docs/BLOCKS.md: what the report holds and what its figures mean. - Test-only: a non-null assertion in arms.test.ts and the sensitivity report strings in methodology.test.ts, now carrying recomputed rates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R6Ax8qvfN3gYUC85oSGnoi

cloudflare-workers-and-pages · 2026-10-12T10:56:10Z

Deploying with Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Updated (UTC)
✅ Deployment successful! View logs	dspack-studio-composer	`e057c44`	Oct 12 2026, 07:28 PM

Copilot

🟡 Changes recommended

Operation aggregation and sensitivity classification can produce inaccurate report conclusions, and below-minimum criteria can still appear as not measurable.

3 open findings

What changed in this PR

Updates Phase 1B measurement correctness without changing frozen scoring or request behavior.

Changes:

Measures whether add/remove refinements perform the requested operation.
Separates harness pacing from execution latency.
Expands sensitivity, sufficiency, parity, path, and failure reporting.

File	Description
`packages/block-bench/live/types.ts`	Adds operation and execution-timing types.
`packages/block-bench/live/report.ts`	Expands measurement reporting.
`packages/block-bench/live/pacing.ts`	Measures actual pacing waits.
`packages/block-bench/live/pacing.test.ts`	Tests wait measurement.
`packages/block-bench/live/operation.ts`	Classifies list-length changes.
`packages/block-bench/live/operation.test.ts`	Tests operation classification.
`packages/block-bench/live/metrics.ts`	Adds sensitivity and revised metrics.
`packages/block-bench/live/metrics.test.ts`	Tests new metric semantics.
`packages/block-bench/live/methodology.test.ts`	Tests reporting and methodology.
`packages/block-bench/live/hosted.test.ts`	Verifies hosted latency separation.
`packages/block-bench/live/dry-run.test.ts`	Checks dry-run operation results.
`packages/block-bench/live/arms.ts`	Records timing and operation measurements.
`packages/block-bench/live/arms.test.ts`	Tests operation recording across arms.
`docs/BLOCKS.md`	Documents revised measurements.

🧠 Review effort: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Tests only; 14 fail on e044e03. - operation.test.ts, arms.test.ts: each list is judged on its own, keyed by where it sits, so a row added to one table never hides a row dropped from another (Copilot). - methodology.test.ts: a verdict that holds while only its clarity rests on missed refinements says so (Copilot, review F3); the token criterion is insufficient below the metered minimum (F4); below the minimum an interval is indicative at most, and a criterion with no interval stays not measurable (F5, Copilot); the report breaks add and remove down by block, counts unchanged misses apart, and says what the pooled rows hold (F1, F6). - metrics.test.ts: unchanged misses and the by-block counts (F1). - pacing.test.ts: waits are measured on a monotonic clock, so a wall clock that jumps during a wait never counts as waiting (F2). - dry-run.test.ts: the refinements table's own row, and the by-block table (F8). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R6Ax8qvfN3gYUC85oSGnoi

…m clarity Report and harness only; the verdict, its reasons and every rule input are unchanged. - operation.ts: each list is keyed by where it sits (a node by its component and its rank among same-component siblings, an array element by its index), so a row added to one table never hides a row dropped from another (Copilot). - metrics.ts, report.ts: the sensitivity reading says whether the verdict or only its clarity rests on refinements that did not change their lists the way their template asked (Copilot, review F3). Misses that left the surface unchanged are counted apart, add and remove are broken down by block, and the line says that an unchanged surface is the right answer where a template does not apply (review F1). - metrics.ts: the token criterion is insufficient below the metered minimum, and so is the decision whenever the verdict's minimum is not met (review F4). Below the minimum an interval is indicative at most; a criterion with no interval stays not measurable (review F5, Copilot). - pacing.ts: waits are measured on the monotonic clock a cell's latency uses, unless a clock is injected (review F2). - report.ts, types.ts, docs/BLOCKS.md: what the pooled rows hold and what execution time still includes (review F6, F7). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R6Ax8qvfN3gYUC85oSGnoi

Tests only; 9 fail on 3abca36. - operation.test.ts, arms.test.ts: a node with an id keeps its lists wherever it moves, so an item put first, a table wrapped in a new container, or reordered siblings are not read as mixed (review minor 1). - methodology.test.ts: the sensitivity line says whether its conclusion survives without the misses that left the surface unchanged (minor 2); the token figure's minimum counts the shared prompts metered in both the block and production arms (nit 3); the report names the plan behind ADR-B3 rather than an undefined "Plan v5" (nit 4). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R6Ax8qvfN3gYUC85oSGnoi

…hout unchanged misses Report and harness only; the verdict, its reasons and every rule input are unchanged. - operation.ts: a node with an id is matched by its component and id wherever it sits, so an item put first, a table moved into a new container, or reordered siblings leave its lists alone; a node without one is still placed by rank among its id-less siblings, and the report says so (review minor 1). - metrics.ts, report.ts: when a miss left its surface unchanged, the sensitivity line also reads the decision with only the misses that changed their surface failing, and says whether its conclusion survives without the unchanged ones (minor 2). - metrics.ts, report.ts: the token criterion's minimum counts the shared prompts metered in both the block and production arms (`meteredCohort`), and the report names that count (nit 3). - report.ts, docs/BLOCKS.md: "the plan behind ADR-B3" for the undefined "Plan v5" (nit 4). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R6Ax8qvfN3gYUC85oSGnoi

Tests only; 12 fail on b0fb8e5. - operation.test.ts: a node is matched by id only where that id is in both surfaces exactly once, so a renamed id, an id on one side only, or a repeated id is matched by place, never read as lists removed and added again or summed into one key (review minor 2, nit 1). - methodology.test.ts: the add and remove sensitivity is one table of readings with a conclusion drawn from all of them, so it can neither call the verdict safe while a reading without the unchanged misses moves it, nor contradict itself (minor 1); a fail clearly missed over the minimum's prompts stays clear whatever the token figure rests on (nit 2). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R6Ax8qvfN3gYUC85oSGnoi

…where unique - operation.ts: a node is matched by its id only where that id occurs exactly once in both surfaces. Any other node is placed by its rank among same-component siblings not matched by id, so a renamed, one-sided or repeated id never reads as lists removed and added again, and never sums two lists into one (review minor 2, nit 1). - metrics.ts: the add and remove sensitivity is a list of readings, one for each way of counting the refinements that did not do what they asked that applies (the missed ones; the missed and mixed ones; and both with only the misses that changed their surface), each recomputed once. verdictHolds, clarityHolds and robust hold over every reading (review minor 1). - metrics.ts: a decision is insufficient only when the prompts every decision arm evaluated, or an arm's threads, fall short of the minimum. A token figure on too few metered prompts stays insufficient as a criterion, so a pass is uncertain and a fail clearly missed stays clear, as the addendum defines a clear decision (review nit 2). - report.ts: the readings are a table under each decision, with a conclusion drawn from all of them; the add and remove note says how nodes are matched. - docs/BLOCKS.md: the same, in the report description. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R6Ax8qvfN3gYUC85oSGnoi

Tests only; 3 fail on d71f2cd. - methodology.test.ts: the report never calls an unchanged surface a correct no-op outright. It is the right answer where a template does not apply, but also what a refinement that ignored its instruction leaves, and a block refinement that re-sends its prior call always expands to the same surface. So the readings that leave unchanged misses out can excuse real ones, and the report says so, beside the decision and under the add and remove table by block (review minor). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R6Ax8qvfN3gYUC85oSGnoi

- report.ts: an unchanged surface is the right answer where a template does not apply, but also what a refinement that ignored its instruction leaves, and a block refinement that re-sends its prior call expands to exactly the same surface. The note beside the readings now says that the readings leaving unchanged misses out can excuse real ones, and the add and remove table by block says that unchanged misses on a template that applies to every surface are real (review minor). - operation.ts, report.ts, docs/BLOCKS.md: matching by id is for nodes in a list, or the root; a node held in a prop of its own is placed by its path (review nit). - metrics.ts: the doc comments say the same. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R6Ax8qvfN3gYUC85oSGnoi

ryandmonk · 2026-10-12T12:50:12Z

Workers Builds: dspack-studio-composer failed on e057c44 (build 85452d1b). The evidence says this is not caused by this PR's changes.

It passed one head earlier. The same check passed on the previous head, d71f2cd, at 12:30 UTC.
The composer build reads none of the changed files.
- The two commits since then, 1e6177f and e057c44, change only docs/BLOCKS.md and four files under packages/block-bench/live/, all strings, comments and a test.
- The composer build (build:css, demo-assets.mjs, next build, check-default-artifact.mjs) reads none of those files, so its inputs at e057c44 are the same as at d71f2cd.
The failing step is unknown. The check run has no output text. The build log is only in the Cloudflare dashboard, which I cannot open, so I cannot see which step failed or re-run the build. There is no fix to port.

Retrying build 85452d1b from the dashboard would settle it. I won't merge while this check is red.

Generated by Claude Code

claude added 2 commits October 12, 2026 10:41

Copilot AI balanced review requested due to automatic review settings October 12, 2026 10:53

Copilot started reviewing on behalf of ryandmonk October 12, 2026 10:53

View session

Copilot AI reviewed Oct 12, 2026

View reviewed changes

Comment thread packages/block-bench/live/metrics.ts

Comment thread packages/block-bench/live/metrics.ts Outdated

Comment thread packages/block-bench/live/operation.ts Outdated

claude added 8 commits October 12, 2026 11:33

ryandmonk merged commit eee21c1 into main Oct 12, 2026
2 checks passed

Navigation Menu

Repository navigation

Phase 1B harness: measure what add and remove did, and keep pacing out of latency - #99

Phase 1B harness: measure what add and remove did, and keep pacing out of latency#99
ryandmonk merged 10 commits into
mainfrom
claude/blocks-measurement-correctness

ryandmonk commented Oct 12, 2026 •

edited

Loading

Uh oh!

cloudflare-workers-and-pages Bot commented Oct 12, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ryandmonk commented Oct 12, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ryandmonk commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes

The frozen boundary

Evidence

Not in this PR

Uh oh!

cloudflare-workers-and-pages Bot commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying with Cloudflare Workers

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ryandmonk commented Oct 12, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ryandmonk commented Oct 12, 2026 •

edited

Loading

cloudflare-workers-and-pages Bot commented Oct 12, 2026 •

edited

Loading