Repository navigation
test: scenarios check a skill's outcome against its contract (ADR 0012) - #52
Merged
Merged
Conversation
glennfaison
force-pushed
the
feat/scenarios
branch
from
November 2, 2026 00:25
a97312b to
330f1b0
Compare
glennfaison
force-pushed
the
fix/contract-gaps
branch
from
November 2, 2026 11:34
1cf0dd6 to
d27f85b
Compare
A scenario is a world, inputs, one result line and the net effects of a run. scripts/scenario.mjs checks it against the skill's real contract: the result is in returns, every effect is in an allowed write scope, invariants hold in the world after (one state label, lease released, merge pinned, keep-open never closed, dry-run writes nothing, escalate and fail have fixed shapes), and an executable eligibility oracle for shepherd-github-pr agrees with the result. 46 scenarios cover all 43 declared results, including the gap fixes. A skill cannot declare a result with no scenario. They check consistency, not that a model follows the skills.
glennfaison
force-pushed
the
feat/scenarios
branch
from
November 2, 2026 11:35
330f1b0 to
9800f4d
Compare
This was referenced Nov 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #51 (base is
fix/contract-gaps, which is stacked on #50). Adds the first evaluation layer on top of the contracts. Design and limits are in ADR 0012.What it is
A scenario is a world (issues and PRs), a skill with its inputs, one expected result line, and the net effects the run leaves behind.
scripts/scenario.mjschecks it against the skill's real contract, read from itsSKILL.md:returns, and is about the target.effects.writesallows.agent:stuckor protected path.skippedleaves nothing,noopleaves only the lease comment,escalatedandfailedhave fixed shapes, dry runs write nothing,limitis respected.shepherd-github-pragrees with the result.What is in it
npm run scenarios:coverage). Every gap fix from fix(skills): resolve the ten contract gaps (ADR 0011) #51 has one, e.g. a conflicted PR with no checks is worked on, and a PR that can only wait on the lane is skipped with no effects at all.tests/scenarios.test.mjs: runs each scenario, plus negative tests that prove the checker rejects broken ones, plus a gate. A skill cannot declare a result line without a scenario. I checked that by adding one and watching the test fail.What it does not prove
shepherd-github-prhas an eligibility oracle.ghsimulator over a world file, and a runner that drives a model against it. The checker is meant to take their output unchanged (diff the world, pass the result line).For review
#N sent backand ofimplement-github-issue's bail-out. Those are the scenarios most worth a second look.