Evaluate the skills as functions: remaining work #72

Category: feature
Area: Evaluation
Impact: Tracks the remaining work to evaluate every skill as a function: a name, parameters, a contract, declared effects, and an outcome that can be checked.
Confidence: traced

Problem (the use case)

A maintainer changing a skill wants to know whether it still does what it promises. #50 gave the pipeline skills contracts, #51 fixed the ten gaps that exposed, and #52 added scenarios that check an outcome against a contract. That checks that the contract, the expected outcomes and the invariants agree. It does not yet check the utility skills, derive effects from a real run, or run a model at all.

Proposed solution

Work through the tickets below. Each is written to stand alone, names its blockers, and says what done means.

1. Checker depth (no model needed, independent of each other)

• Write the eligibility rule of the other item skills as code #64 Write the eligibility rule of the other item skills as code (small)
• Check that running a skill a second time does nothing #65 Check that running a skill a second time does nothing (medium, after the oracles)
• Check that group skills select and dispatch what they should #66 Check that group skills select and dispatch what they should (medium)
• Refactor the effects.writes entries to be checkable, not just prose #67 Refactor the effects.writes entries to be checkable, not just prose (large, split in three)

2. Utility and meta skills (start with the first)

• Check utility and meta skills against a contract, as well as the pipeline steps #53 Check utility and meta skills against a contract (medium, blocks the five below)
• Check that merge-safety returns a verdict and changes nothing #54 merge-safety (small)
• Check that unattended-triage returns a recommended state and comment, and does not write #55 unattended-triage (small)
• Check that grill-via-github-comments asks exactly one round, each question with a recommended answer #56 grill-via-github-comments (small)
• Check that retrospective reads only the log window it was given and ranks at most max_suggestions #57 retrospective (medium)
• Check that setup-github-agent-workflow creates only the labels and settings it declares #58 setup-github-agent-workflow (medium)

3. Judging a real run (this needs a model, and the order matters)

• Turn the difference between two worlds into effects #59 Turn the difference between two worlds into effects (small)
• Run a skill against a simulated GitHub instead of the real one #60 Run a skill against a simulated GitHub (large, split in three)
• Run a model through a skill against the simulator and check the outcome #61 Run a model through a skill and check the outcome (medium)
• Measure how consistently a model follows a skill #62 Measure how consistently a model follows a skill (medium)
• Confirm or correct the golden scenarios that rest on inferred behaviour #63 Confirm or correct the golden scenarios that rest on inferred behaviour (debt, after the runner)

4. Design

• Let implement-github-issue split an issue that is too large into sub-issues #68 Let implement-github-issue split an issue that is too large (large, ADR first)
• One lease comment per claim costs a comment on every item on every run #69 One lease comment per claim costs a comment on every item on every run (debt)

5. Housekeeping (small, no blockers)

• The runner docs do not mention the sweeper, write_log, lease_held or limit #70 The runner docs do not mention the sweeper and the new inputs
• npm test fails with 65 shell-test errors, and no clear message, when jq is missing #71 npm test fails without jq, with no clear message

Suggested order

1. Housekeeping, in parallel with anything.
2. The utility-skill ticket and oracles, then their dependants. These need no model and show whether the contract format generalises.
3. diff, then sim, then runner. Only the runner makes the evaluation real, and verify and consist depend on it.
4. structured when the checker's scope-only limit starts to cost something. recurse is independent and needs an ADR before any code.

Acceptance criteria

• Every ticket above is closed or explicitly dropped with a reason.
• npm run eval judges a real run of each of the five item skills.
• Every skill in skills/ has a contract, or is explicitly a reference skill.

Non-goals

Changing what any skill does, except where a ticket above says so.