Skip to content

test: scenarios check a skill's outcome against its contract (ADR 0012) - #52

Merged
glennfaison merged 1 commit into
mainfrom
feat/scenarios
Nov 2, 2026
Merged

glennfaison merged 1 commit into
mainfrom
feat/scenarios

Conversation

@glennfaison

Copy link
Copy Markdown
Owner

Stacked on #51 (base is fix/contract-gaps, which is stacked on #50). Adds the first evaluation layer on top of the contracts. Design and limits are in ADR 0012.

What it is

A scenario is a world (issues and PRs), a skill with its inputs, one expected result line, and the net effects the run leaves behind. scripts/scenario.mjs checks it against the skill's real contract, read from its SKILL.md:

  1. The result is one of returns, and is about the target.
  2. Every effect is in a scope that effects.writes allows.
  3. The world after the effects satisfies the invariants: one state label per issue, one PR state label per PR, one PR on the merge lane, lease released, keep-open never closed, merge pinned to the reviewed head and never in a run that pushed, never a fork, draft, agent:stuck or protected path. skipped leaves nothing, noop leaves only the lease comment, escalated and failed have fixed shapes, dry runs write nothing, limit is respected.
  4. An executable eligibility oracle for shepherd-github-pr agrees with the result.

What is in it

  • 46 golden scenarios, covering all 43 declared results (npm run scenarios:coverage). Every gap fix from fix(skills): resolve the ten contract gaps (ADR 0011) #51 has one, e.g. a conflicted PR with no checks is worked on, and a PR that can only wait on the lane is skipped with no effects at all.
  • tests/scenarios.test.mjs: runs each scenario, plus negative tests that prove the checker rejects broken ones, plus a gate. A skill cannot declare a result line without a scenario. I checked that by adding one and watching the test fail.
  • 137 node tests and 154 shell tests pass, with lint, validate and sync.

What it does not prove

  • It does not run a skill. There is no model in CI, and none in my sandbox. The scenarios check that the contract, the expected outcomes and the invariants agree with each other. They do not show that a model follows the skill text.
  • I wrote the goldens and the contracts from the same reading of the skills, so they can be wrong together. A real run that contradicts one should change one of them.
  • Scopes are checked, not the exact prose under them. Only shepherd-github-pr has an eligibility oracle.
  • The pieces that would judge a real run are not built: a gh simulator over a world file, and a runner that drives a model against it. The checker is meant to take their output unchanged (diff the world, pass the result line).

For review

  • A few expected effects rest on my reading of the skill text where it was not explicit, for example the effects of #N sent back and of implement-github-issue's bail-out. Those are the scenarios most worth a second look.

A scenario is a world, inputs, one result line and the net effects of a run.
scripts/scenario.mjs checks it against the skill's real contract: the result is
in returns, every effect is in an allowed write scope, invariants hold in the
world after (one state label, lease released, merge pinned, keep-open never
closed, dry-run writes nothing, escalate and fail have fixed shapes), and an
executable eligibility oracle for shepherd-github-pr agrees with the result.

46 scenarios cover all 43 declared results, including the gap fixes. A skill
cannot declare a result with no scenario. They check consistency, not that a
model follows the skills.
@glennfaison
glennfaison changed the base branch from fix/contract-gaps to main November 2, 2026 11:35
@glennfaison
glennfaison merged commit 9234b75 into main Nov 2, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants