diff --git a/AGENTS.md b/AGENTS.md index 529653c..5f3bbdb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -35,6 +35,8 @@ skills///SKILL.md exactly three levels deep **Every pipeline skill opens with a `## Contract`**: a YAML block that says what the skill is as a function (preconditions, the closed set of results, postconditions, side effects by scope, forbidden effects). **Process** says how. Change a result line, a precondition or an effect in the contract first. `agent-workflow-protocol` defines the keys (**Contracts**), `docs/adr/0010-skills-as-functions-with-contracts.md` explains why, and `npm run validate` checks that a contract matches the rest of its skill: `params` equal the Inputs table, every result the text returns is in `returns`, and there are no `Eligibility`, `Outputs`, `Result` or `Never` sections left over. Anything not listed under `effects.writes` is forbidden. +**Every result a skill can return has a scenario** under `tests/scenarios//`: a world, the inputs, the expected result line and the net effects. `scripts/scenario.mjs` checks each against the skill's contract (scopes, invariants, eligibility oracles), and `npm run scenarios:coverage` lists results with none. A new result line without a scenario fails `npm test`. See `docs/adr/0012-scenarios.md`. + ## Inputs and defaults **Every skill has an `## Inputs` section.** It names the inputs the skill accepts, in two interchangeable forms: **key-value pairs** (`key=value` on a line of its own) and **free text** (whatever the prompt says). It also names the skill's **implicit dependencies**: the tools, skills and repository state it needs that are not inputs. Every pipeline skill lists `runner` and `unattended`, and every skill that reads markers lists `trusted_authors`. diff --git a/docs/adr/0012-scenarios.md b/docs/adr/0012-scenarios.md new file mode 100644 index 0000000..6fefc7a --- /dev/null +++ b/docs/adr/0012-scenarios.md @@ -0,0 +1,38 @@ +# 0012. Scenarios check a skill's outcome against its contract + +## Status + +Accepted + +## Context + +ADR 0010 gave every pipeline skill a contract, and ADR 0011 closed the gaps it exposed. A contract says what a skill promises. Nothing yet checked an outcome against it. The skills are instructions a model follows, so a test cannot run one without a model, and no model is available in CI. + +## Decision + +A **scenario** is a world, a skill and its inputs, and the expected outcome: one result line, and the net effects the run leaves behind (`tests/scenarios//*.yaml`). `scripts/scenario.mjs` checks a scenario against the skill's real contract, read from its `SKILL.md`: + +1. The result is one of the contract's `returns`, and is about the target. +2. Every effect is in a scope that `effects.writes` allows (`claimed`, `repository`, `other`, `shared`). +3. The world after the effects satisfies the invariants: + - one state label per issue, one PR state label per PR, one PR holding the merge lane; + - the lease label is gone from the claimed item; + - no keep-open issue is closed; + - a merge is pinned to the reviewed head, is not in a run that pushed, and is not of a fork, a draft, an `agent:stuck` PR or a protected path; + - `skipped` leaves no effects, `noop` leaves only the lease comment, `escalated` adds `agent:stuck` with an `agent:escalate` comment, `failed` leaves the labels alone; + - a dry run writes nothing, `limit` is respected, and only skills in the contract's `calls` are called. +4. Eligibility **oracles**, which are `requires` written as code, agree with the result. `shepherd-github-pr` has one. + +`tests/scenarios.test.mjs` runs every scenario, runs negative tests that prove the checker rejects broken ones, and requires every result a skill declares to have at least one scenario (`npm run scenarios:coverage`). + +## Consequences + +- A new result line, or a changed write scope, cannot land without a scenario. The old behaviour of the gap-9 bug (a conflicted PR skipped as `no-checks-yet`) is now a failing test. +- The same checker can judge a real run. Diff the world before and after, turn the diff into effects, and pass the result line. The part that is not built is the part that needs infrastructure: a `gh` simulator over a world file, and a runner that drives a model against it. + +## What this does not prove + +- **It does not run a skill.** The goldens are written from the skill text, so they check that the contract, the expected outcomes and the invariants agree with each other. They cannot show that a model follows the text. +- **The goldens can be wrong in the same way as the contract.** Both came from the same reading of the skills. A scenario that a real run contradicts should change one of them. +- **Scopes are checked, not the lists under them.** The contract's `effects.writes` entries are prose, so an effect is checked against its scope and not against the exact sentence. +- **Only one eligibility oracle exists.** Writing `requires` as code for the other skills duplicates the prose, so each one is a judgement about whether the check is worth keeping in step. diff --git a/package.json b/package.json index 6907147..7a88839 100644 --- a/package.json +++ b/package.json @@ -8,7 +8,7 @@ "validate": "node scripts/validate-skills.mjs", "lint": "node scripts/lint-files.mjs --all", "pretest": "node scripts/check-hooks.mjs", - "test": "bash tests/lib.test.sh && node --test tests/helpers.test.mjs tests/lint.test.mjs tests/contracts.test.mjs", + "test": "bash tests/lib.test.sh && node --test tests/helpers.test.mjs tests/lint.test.mjs tests/contracts.test.mjs tests/scenarios.test.mjs", "check-plugin-version": "node scripts/sync-plugin-version.mjs --check", "check": "npm run lint && npm run validate && npm run check-plugin-version && npm test", "hooks:check": "node scripts/check-hooks.mjs", @@ -17,7 +17,8 @@ "changeset": "changeset", "version": "changeset version && node scripts/sync-plugin-version.mjs", "sync": "node scripts/sync.mjs", - "sync:check": "node scripts/sync.mjs --check" + "sync:check": "node scripts/sync.mjs --check", + "scenarios:coverage": "node scripts/scenario-coverage.mjs" }, "dependencies": { "mattpocock-skills": "1.3.0" diff --git a/scripts/scenario-coverage.mjs b/scripts/scenario-coverage.mjs new file mode 100644 index 0000000..f348d56 --- /dev/null +++ b/scripts/scenario-coverage.mjs @@ -0,0 +1,40 @@ +// Reports which of each skill's declared results no scenario exercises. +// npm run scenarios:coverage +// The same numbers are enforced by tests/scenarios.test.mjs: a skill cannot declare a result line +// that no scenario exercises. +import { readFileSync, readdirSync, existsSync } from "node:fs"; +import { join, dirname } from "node:path"; +import { fileURLToPath, pathToFileURL } from "node:url"; +import { parse } from "yaml"; +import { extractContract } from "./contracts.mjs"; +import { matchesReturn } from "./scenario.mjs"; + +const PKG = join(dirname(fileURLToPath(import.meta.url)), ".."); +const skills = readdirSync(join(PKG, "skills/pipeline")).filter((n) => existsSync(join(PKG, "skills/pipeline", n, "SKILL.md"))).sort(); +/** @returns {{name: string, total: number, missing: string[]}[]} one row per non-group pipeline skill */ +export function coverage() { + const rows = []; + for (const name of skills) { + const c = parse(extractContract(readFileSync(join(PKG, "skills/pipeline", name, "SKILL.md"), "utf8")).yaml); + if (c.kind === "group") continue; // a group's results are its items' results + const dir = join(PKG, "tests/scenarios", name); + const results = existsSync(dir) + ? readdirSync(dir).filter((f) => f.endsWith(".yaml")).flatMap((f) => { + const sc = parse(readFileSync(join(dir, f), "utf8")); + return (sc.expect.results ?? [sc.expect.result]).flatMap((r) => [r, r.replace(/^would: /, "")]); + }) + : []; + rows.push({ name, total: c.returns.length, missing: c.returns.filter((r) => !results.some((x) => matchesReturn(x, [r]))) }); + } + return rows; +} + +if (import.meta.url === pathToFileURL(process.argv[1] ?? "").href) { + let total = 0, covered = 0; + for (const { name, total: t, missing } of coverage()) { + total += t; covered += t - missing.length; + console.log(`${name.padEnd(26)} ${String(t - missing.length).padStart(2)}/${t} results covered`); + for (const m of missing) console.log(` not covered: ${m}`); + } + console.log(`\n${covered}/${total} declared results have a scenario`); +} diff --git a/scripts/scenario.mjs b/scripts/scenario.mjs new file mode 100644 index 0000000..cea1d1f --- /dev/null +++ b/scripts/scenario.mjs @@ -0,0 +1,246 @@ +// Checks a scenario against a skill's contract. Pure functions, no GitHub and no model. +// +// A scenario says: this skill, these inputs, this world. The expected result is one line, and the +// expected effects are the net changes the run leaves behind. The checker then asks four things: +// 1. Is the result one of the contract's `returns`? +// 2. Is every effect in a scope the contract's `effects.writes` allows? +// 3. Do the invariants hold in the world after the effects (one state label, lease released, ...)? +// 4. Do the executable eligibility oracles agree with the result? +// +// It does not run a skill. A real run is checked the same way: diff the world before and after, +// turn the diff into effects, and pass the result line. See docs/adr/0012-scenarios.md. + +export const MUTEX_ISSUE = ["needs-info", "ready-for-agent", "ready-for-human", "agent:pr-open"]; +export const MUTEX_PR = ["agent:needs-work", "agent:ready-for-review", "agent:human-merge"]; +export const LEASE_LABELS = ["agent:triage", "agent:in-progress"]; +const REPOSITORY_OPS = ["push", "merge", "delete_branch", "create_pr"]; +const OPS = ["label_add", "label_remove", "comment", "close", "push", "merge", "delete_branch", "create_pr", "create_issue", "review_request"]; + +/** "pr#40" -> { kind: "pr", n: 40 } */ +export function ref(t) { + const m = /^(issue|pr)#(\d+)$/.exec(String(t)); + if (!m) throw new Error(`bad target \`${t}\`: expected issue#N or pr#N`); + return { kind: m[1], n: Number(m[2]) }; +} + +export function item(world, target) { + const { kind, n } = ref(target); + return (kind === "issue" ? world.issues : world.prs)?.find((x) => x.number === n); +} + +/** The merge-lane holder: another open PR carrying agent:ready-for-review, or null. */ +export function laneHolder(world, n) { + return world.prs?.find((p) => p.number !== n && p.state === "open" && (p.labels ?? []).includes("agent:ready-for-review")) ?? null; +} + +/** "no-checks-yet" | "pending" | "failing" | "green", as shepherd-github-pr reads it. */ +export function checkState(pr) { + const c = pr.checks ?? []; + if (c.length === 0) return "no-checks-yet"; + if (c.includes("pending")) return "pending"; + if (c.includes("fail") || c.includes("cancel")) return "failing"; + return "green"; +} + +// ---- oracles: eligibility written as code, from the skill's `requires` ------------------------- + +export const ORACLES = { + /** Mirrors the `requires` of shepherd-github-pr. Returns { eligible, reason }. */ + "shepherd-github-pr"(world, target) { + const pr = item(world, target); + const L = pr.labels ?? []; + if (pr.state !== "open" || !L.includes("agent:needs-work") || L.includes("agent:stuck") || L.includes("agent:human-merge")) { + return { eligible: false, reason: "not an open agent:needs-work PR" }; + } + const checks = checkState(pr); + const conflict = pr.mergeState === "DIRTY"; + if (!(checks === "failing" || checks === "green" || conflict)) return { eligible: false, reason: `checks ${checks}` }; + const needsFix = checks === "failing" || (pr.openFindings ?? 0) > 0 || conflict; + if (!needsFix) { + const holder = laneHolder(world, pr.number); + if (holder) return { eligible: false, reason: `merge lane held by #${holder.number}` }; + if (pr.mergeState === "UNKNOWN") return { eligible: false, reason: "merge state not computed" }; + } + return { eligible: true }; + }, +}; + +// ---- effects ----------------------------------------------------------------------------------- + +/** Which scope of `effects.writes` an effect falls in. */ +export function scopeOf(effect, ctx) { + if (REPOSITORY_OPS.includes(effect.op)) return "repository"; + if (effect.op === "comment") { + const t = item(ctx.world, effect.target); + if (t && (t.labels ?? []).includes("agent:log")) return "shared"; + } + if (ctx.claimed.has(effect.target) && effect.op !== "create_issue") return "claimed"; + return "other"; +} + +/** Apply effects to a copy of the world. Throws when an effect is impossible. */ +export function applyEffects(world, effects) { + const w = structuredClone(world); + w.issues ??= []; w.prs ??= []; + for (const e of effects) { + if (!OPS.includes(e.op)) throw new Error(`unknown effect op \`${e.op}\``); + if (e.op === "create_pr") { w.prs.push({ number: e.number, state: "open", labels: e.labels ?? [], draft: false, head: e.sha ?? "new", base: e.base ?? w.default_branch ?? "main", checks: [], openFindings: 0, closes: e.closes ?? [], comments: [] }); continue; } + if (e.op === "create_issue") { w.issues.push({ number: e.number, state: "open", labels: e.labels ?? [], comments: [] }); continue; } + const t = item(w, e.target); + if (!t) throw new Error(`effect ${e.op} targets ${e.target}, which is not in the world`); + t.labels ??= []; t.comments ??= []; + switch (e.op) { + case "label_add": if (!t.labels.includes(e.label)) t.labels.push(e.label); break; + case "label_remove": + if (!t.labels.includes(e.label)) throw new Error(`removes \`${e.label}\` from ${e.target}, which does not carry it`); + t.labels = t.labels.filter((l) => l !== e.label); break; + case "comment": t.comments.push({ marker: e.marker, text: e.text }); break; + case "close": t.state = "closed"; break; + case "push": t.head = e.sha ?? `${t.head}+`; t.pushed = true; break; + case "merge": t.state = "merged"; break; + case "delete_branch": t.branchDeleted = true; break; + case "review_request": break; + } + } + return w; +} + +export function matchesReturn(line, returns) { + return returns.some((r) => { + const re = r.replace(/[.*+?^${}()|[\]\\]/g, "\\$&").replace(/<[^>]*>/g, ".+").replace(/#[NPM]\b/g, "#\\d+"); + return new RegExp(`^${re}$`).test(line); + }); +} + +const head = (line) => line.match(/^#\d+\s+([A-Za-z][A-Za-z-]*)/)?.[1] ?? null; + +/** + * @param {object} contract the parsed contract of the skill + * @param {object} sc the scenario + * @returns {string[]} problems, empty when the scenario is consistent with the contract + */ +export function checkScenario(contract, sc) { + const out = []; + const bad = (m) => out.push(m); + const world = sc.world ?? { issues: [], prs: [] }; + const results = sc.expect.results ?? (sc.expect.result ? [sc.expect.result] : []); + const work = sc.expect.effects ?? []; + const isItem = contract.kind === "item" && contract.target?.param; + const claimed = new Set(isItem && sc.target ? [sc.target] : []); + const leaseHeld = sc.inputs?.lease_held === true; + + // 1. results + if (results.length === 0) bad("no expected result"); + for (const r of results) { + const line = r.replace(/^would: /, ""); + if (!matchesReturn(r, contract.returns) && !matchesReturn(line, contract.returns)) bad(`result \`${r}\` is not in the contract's returns`); + if (isItem && !line.startsWith(`#${ref(sc.target).n} `)) bad(`result \`${r}\` is not about ${sc.target}`); + } + const outcome = isItem && results[0] ? head(results[0]) : null; + + // effects, with the claim's lease marker made explicit unless the run was skipped or borrowed a lease + const effects = [...work]; + if (isItem && outcome !== "skipped" && !leaseHeld) effects.unshift({ op: "comment", target: sc.target, marker: "agent:lease" }); + const ctx = { world, claimed }; + + // 2. scopes + const allowed = Object.keys(contract.effects?.writes ?? {}); + for (const e of work) { + let scope; + try { scope = scopeOf(e, ctx); } catch (err) { bad(err.message); continue; } + if (!allowed.includes(scope)) bad(`${e.op} on ${e.target ?? `#${e.number}`} is a \`${scope}\` write, which the contract does not allow`); + } + if (outcome === "skipped" && work.length) bad("skipped must leave no effects"); + if (outcome === "noop" && work.length) bad("noop may leave only the lease comment"); + + // 3. the world after + let after; + try { after = applyEffects(world, effects); } catch (err) { bad(err.message); return out; } + for (const i of after.issues) { + if (i.lease === "live") continue; // another step holds it, and the sweeper must not repair it (labels.md, Repair) + const n = MUTEX_ISSUE.filter((l) => (i.labels ?? []).includes(l)); + if (n.length > 1) bad(`issue #${i.number} ends with ${n.join(" and ")}`); + } + for (const p of after.prs) { + const n = MUTEX_PR.filter((l) => (p.labels ?? []).includes(l)); + if (n.length > 1) bad(`PR #${p.number} ends with ${n.join(" and ")}`); + } + const lanes = after.prs.filter((p) => p.state === "open" && (p.labels ?? []).includes("agent:ready-for-review")); + if (lanes.length > 1) bad(`${lanes.length} PRs hold the merge lane: ${lanes.map((p) => `#${p.number}`).join(", ")}`); + if (isItem && !leaseHeld) { + const t = item(after, sc.target); + for (const l of LEASE_LABELS) if ((t.labels ?? []).includes(l)) bad(`${sc.target} still carries the lease label ${l}`); + } + for (const e of work) { + if (e.op === "close") { + const t = item(world, e.target); + if (t?.keepOpen) bad(`closes ${e.target}, which is keep-open`); + } + } + + // merge rules (review-github-pr) + for (const e of work.filter((x) => x.op === "merge")) { + const pr = item(world, e.target); + if (e.sha !== pr.head) bad(`merges ${e.target} at ${e.sha}, but the pinned head is ${pr.head}`); + if (work.some((x) => x.op === "push" && x.target === e.target)) bad(`merges ${e.target} in the run that pushed to it`); + if (pr.fork || pr.draft) bad(`merges ${e.target}, which is a ${pr.fork ? "fork" : "draft"}`); + for (const l of ["agent:stuck", "agent:human-merge"]) if ((pr.labels ?? []).includes(l)) bad(`merges ${e.target}, which carries ${l}`); + const prot = sc.inputs?.human_merge_paths ?? []; + const hit = (pr.files ?? []).find((f) => prot.some((p) => f.startsWith(p))); + if (hit) bad(`merges ${e.target}, which changes the protected path ${hit}`); + } + + // 4. oracles + const oracle = ORACLES[sc.skill]; + if (oracle && isItem) { + // a `noop` is a PR that was eligible when read and not after the claim, so the oracle reads the earlier world + const o = oracle(sc.eligibility_world ?? world, sc.target); + if (!o.eligible && outcome !== "skipped") bad(`the oracle says ${sc.target} is not eligible (${o.reason}), but the result is \`${results[0]}\``); + if (o.eligible && outcome === "skipped") bad(`the oracle says ${sc.target} is eligible, but the result is \`${results[0]}\``); + } + if (sc.skill === "shepherd-github-pr" && isItem) { + const added = work.some((e) => e.op === "label_add" && e.label === "agent:ready-for-review"); + const p = item(after, sc.target); + if (added && !(checkState(p) === "green" && !["DIRTY", "BEHIND", "UNKNOWN"].includes(p.mergeState) && (p.openFindings ?? 0) === 0)) { + bad("hands off with agent:ready-for-review while the PR is not green, current and free of findings"); + } + if (added !== (outcome === "ready-for-review")) bad("`ready-for-review` and the label swap must go together"); + } + // escalation and failure have fixed shapes (references/escalate.md, fail_item in lib.sh) + if (isItem && outcome === "escalated") { + if (!work.some((e) => e.op === "comment" && e.target === sc.target && e.marker === "agent:escalate")) bad("escalates without an `agent:escalate` comment on the item"); + if (!work.some((e) => e.op === "label_add" && e.target === sc.target && e.label === "agent:stuck")) bad("escalates without adding agent:stuck"); + } + if (isItem && outcome === "failed") { + if (!work.some((e) => e.op === "comment" && e.target === sc.target && e.marker === "agent:failure")) bad("fails without an `agent:failure` comment on the item"); + if (work.some((e) => (e.op === "label_add" || e.op === "label_remove") && e.target === sc.target)) bad("a failed run must leave the item's labels as they were"); + } + + // dry runs write nothing + const dry = sc.inputs?.dry_run === true || results.some((r) => r.startsWith("would: ")); + if (dry && work.length) bad("a dry run must leave no effects"); + if (sc.inputs?.dry_run === true && sc.skill !== "file-github-issues" && results.some((r) => r !== "nothing to do" && !r.startsWith("would: "))) { + bad("with dry_run=true every result line starts with `would: `"); + } + + // limit (file-github-issues): at most `limit` filed, and nothing is deferred before the limit is reached + if (sc.inputs?.limit !== undefined) { + const filed = results.filter((r) => /^(filed #|would-file: )/.test(r)).length; + const deferred = results.filter((r) => r.startsWith("deferred: ")).length; + if (filed > sc.inputs.limit) bad(`files ${filed} candidates with limit=${sc.inputs.limit}`); + if (deferred && filed < sc.inputs.limit) bad("defers candidates before the limit was reached"); + } + + // skills called must be in the contract's `calls` + for (const c of sc.expect.calls ?? []) if (!(contract.calls ?? []).includes(c)) bad(`calls \`${c}\`, which is not in the contract's calls`); + + // triage-github-issue: a prior trusted outcome that stands is reused, so unattended-triage is not called for it + if (sc.skill === "triage-github-issue" && sc.target) { + const t = item(world, sc.target); + const cs = t.comments ?? []; + const i = cs.findLastIndex((c) => c.marker === "agent:outcome" && c.trusted); + const stands = i !== -1 && !cs.slice(i + 1).some((c) => c.author === "human"); + if (stands && (sc.expect.calls ?? []).includes("unattended-triage")) bad("calls unattended-triage although a prior trusted outcome still stands"); + } + return out; +} diff --git a/tests/scenarios.test.mjs b/tests/scenarios.test.mjs new file mode 100644 index 0000000..0cc0326 --- /dev/null +++ b/tests/scenarios.test.mjs @@ -0,0 +1,173 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { readFileSync, readdirSync } from "node:fs"; +import { join, dirname } from "node:path"; +import { fileURLToPath } from "node:url"; +import { parse } from "yaml"; +import { extractContract } from "../scripts/contracts.mjs"; +import { coverage } from "../scripts/scenario-coverage.mjs"; +import { checkScenario, applyEffects, matchesReturn, checkState, ORACLES } from "../scripts/scenario.mjs"; + +const PKG = join(dirname(fileURLToPath(import.meta.url)), ".."); +const contractOf = (skill) => { + const body = readFileSync(join(PKG, "skills/pipeline", skill, "SKILL.md"), "utf8"); + return parse(extractContract(body).yaml); +}; +const dir = join(PKG, "tests/scenarios"); +const scenarios = readdirSync(dir).flatMap((skill) => + readdirSync(join(dir, skill)).filter((f) => f.endsWith(".yaml")).map((f) => ({ file: `${skill}/${f}`, sc: parse(readFileSync(join(dir, skill, f), "utf8")) }))); + +// ---- every golden scenario agrees with its skill's real contract ------------------------------ +for (const { file, sc } of scenarios) { + test(`scenario ${file}: ${sc.name}`, () => { + assert.equal(file.split("/")[0], sc.skill, "the scenario sits in its skill's directory"); + assert.deepEqual(checkScenario(contractOf(sc.skill), sc), []); + }); +} + +// ---- every result a skill can return has a scenario ------------------------------------------- +test("every declared result has a scenario", () => { + const holes = coverage().flatMap((r) => r.missing.map((m) => `${r.name}: ${m}`)); + assert.deepEqual(holes, [], "add a scenario under tests/scenarios// for each"); +}); + +// ---- the checker catches broken scenarios ------------------------------------------------------ +const shepherd = contractOf("shepherd-github-pr"); +const review = contractOf("review-github-pr"); +const reconcile = contractOf("reconcile-github-issues"); +const find = (name) => structuredClone(scenarios.find((s) => s.file === name).sc); +const problems = (contract, sc) => checkScenario(contract, sc); + +test("a result that is not in returns is rejected", () => { + const sc = find("shepherd-github-pr/pending-checks-are-skipped.yaml"); + sc.expect.result = "#40 exploded"; + assert.ok(problems(shepherd, sc).some((p) => /not in the contract's returns/.test(p))); +}); + +test("a skipped run with effects is rejected", () => { + const sc = find("shepherd-github-pr/pending-checks-are-skipped.yaml"); + sc.expect.effects = [{ op: "comment", target: "pr#40" }]; + assert.ok(problems(shepherd, sc).some((p) => /skipped must leave no effects/.test(p))); +}); + +test("a noop that changes anything is rejected", () => { + const sc = find("shepherd-github-pr/lane-taken-after-the-claim.yaml"); + sc.expect.effects = [{ op: "label_add", target: "pr#40", label: "agent:ready-for-review" }]; + assert.ok(problems(shepherd, sc).some((p) => /noop may leave only the lease comment/.test(p))); +}); + +test("the oracle rejects skipping a conflicted PR that has no checks", () => { + const sc = find("shepherd-github-pr/conflict-without-checks.yaml"); + sc.expect.result = "#40 skipped: checks no-checks-yet"; + sc.expect.effects = []; + assert.ok(problems(shepherd, sc).some((p) => /oracle says pr#40 is eligible/.test(p))); +}); + +test("the oracle rejects working a PR that can only wait on the lane", () => { + const sc = find("shepherd-github-pr/lane-held-is-skipped-without-a-trace.yaml"); + sc.expect.result = "#40 ready-for-review"; + sc.expect.effects = [ + { op: "label_remove", target: "pr#40", label: "agent:needs-work" }, + { op: "label_add", target: "pr#40", label: "agent:ready-for-review" }, + ]; + const p = problems(shepherd, sc); + assert.ok(p.some((x) => /not eligible \(merge lane held by #41\)/.test(x))); + assert.ok(p.some((x) => /2 PRs hold the merge lane/.test(x))); +}); + +test("a hand-off while checks are not green is rejected", () => { + const sc = find("shepherd-github-pr/green-and-lane-free-hands-off.yaml"); + sc.world.prs[0].checks = ["fail"]; + assert.ok(problems(shepherd, sc).some((p) => /not green, current and free of findings/.test(p))); +}); + +test("a PR ending in two PR state labels is rejected", () => { + const sc = find("shepherd-github-pr/green-and-lane-free-hands-off.yaml"); + sc.expect.effects = sc.expect.effects.filter((e) => e.op !== "label_remove"); + assert.ok(problems(shepherd, sc).some((p) => /ends with agent:needs-work and agent:ready-for-review/.test(p))); +}); + +test("a lease label left behind is rejected", () => { + const sc = find("shepherd-github-pr/green-and-lane-free-hands-off.yaml"); + sc.world.prs[0].labels.push("agent:in-progress"); + assert.ok(problems(shepherd, sc).some((p) => /still carries the lease label agent:in-progress/.test(p))); +}); + +test("closing a keep-open issue is rejected", () => { + const sc = find("reconcile-github-issues/keep-open-stays-open.yaml"); + sc.expect.effects = [{ op: "label_remove", target: "issue#12", label: "agent:pr-open" }, { op: "close", target: "issue#12" }]; + assert.ok(problems(reconcile, sc).some((p) => /keep-open/.test(p))); +}); + +test("the sweeper may not claim: its effects are `other`, and a repository write is refused", () => { + const sc = find("reconcile-github-issues/merged-pr-closes-the-issue.yaml"); + sc.expect.effects.push({ op: "push", target: "pr#40" }); + assert.ok(problems(reconcile, sc).some((p) => /`repository` write/.test(p))); +}); + +test("a dry run that writes is rejected", () => { + const sc = find("reconcile-github-issues/dry-run-writes-nothing.yaml"); + sc.expect.effects = [{ op: "close", target: "issue#12" }]; + assert.ok(problems(reconcile, sc).some((p) => /dry run must leave no effects/.test(p))); +}); + +test("merging a moved head, a pushed PR, a draft or a protected path is rejected", () => { + const base = () => find("review-github-pr/merge-closes-the-linked-issue.yaml"); + let sc = base(); sc.expect.effects[0].sha = "other"; + assert.ok(problems(review, sc).some((p) => /pinned head is h1/.test(p))); + sc = base(); sc.expect.effects.push({ op: "push", target: "pr#50" }); + assert.ok(problems(review, sc).some((p) => /run that pushed/.test(p))); + sc = base(); sc.world.prs[0].draft = true; + assert.ok(problems(review, sc).some((p) => /draft/.test(p))); + sc = base(); sc.inputs = { human_merge_paths: ["src/"] }; + assert.ok(problems(review, sc).some((p) => /protected path src\/a.js/.test(p))); +}); + +test("filing past the limit, or deferring before it, is rejected", () => { + const file = contractOf("file-github-issues"); + let sc = find("file-github-issues/limit-defers-the-rest.yaml"); + sc.inputs.limit = 1; + assert.ok(problems(file, sc).some((p) => /files 2 candidates with limit=1/.test(p))); + sc = find("file-github-issues/limit-defers-the-rest.yaml"); + sc.inputs.limit = 3; + assert.ok(problems(file, sc).some((p) => /before the limit was reached/.test(p))); +}); + +test("calling unattended-triage when a trusted outcome stands is rejected", () => { + const triage = contractOf("triage-github-issue"); + const sc = find("triage-github-issue/a-standing-prior-outcome-is-reused.yaml"); + sc.expect.calls = ["unattended-triage"]; + assert.ok(problems(triage, sc).some((p) => /although a prior trusted outcome still stands/.test(p))); +}); + +test("a borrowed lease means no lease comment and no release is expected", () => { + const sc = find("shepherd-github-pr/green-and-lane-free-hands-off.yaml"); + sc.inputs = { lease_held: true }; + sc.world.prs[0].labels.push("agent:in-progress"); + assert.deepEqual(problems(shepherd, sc), []); +}); + +test("escalating without agent:stuck, or failing while changing labels, is rejected", () => { + const tri = contractOf("triage-github-issue"); + let sc = find("triage-github-issue/repeated-failures-escalate.yaml"); + sc.expect.effects = sc.expect.effects.filter((e) => e.label !== "agent:stuck"); + assert.ok(problems(tri, sc).some((p) => /without adding agent:stuck/.test(p))); + sc = find("triage-github-issue/a-failed-pass-leaves-the-state.yaml"); + sc.expect.effects.push({ op: "label_add", target: "issue#12", label: "needs-info" }); + assert.ok(problems(tri, sc).some((p) => /labels as they were/.test(p))); + sc = find("triage-github-issue/a-failed-pass-leaves-the-state.yaml"); + sc.expect.effects = []; + assert.ok(problems(tri, sc).some((p) => /without an `agent:failure` comment/.test(p))); +}); + +test("helpers", () => { + assert.equal(checkState({ checks: [] }), "no-checks-yet"); + assert.equal(checkState({ checks: ["pass", "pending"] }), "pending"); + assert.equal(checkState({ checks: ["pass", "fail"] }), "failing"); + assert.equal(checkState({ checks: ["pass"] }), "green"); + assert.ok(matchesReturn("#40 noop: merge lane held by #41", ["#N noop: "])); + assert.ok(matchesReturn("#50 PR #51 opened", ["#N PR #P opened"])); + assert.ok(!matchesReturn("#50 merged now", ["#N merged"])); + assert.throws(() => applyEffects({ issues: [{ number: 1, labels: [] }], prs: [] }, [{ op: "label_remove", target: "issue#1", label: "x" }]), /does not carry it/); + assert.ok(ORACLES["shepherd-github-pr"]); +}); diff --git a/tests/scenarios/file-github-issues/a-duplicate-gets-a-comment.yaml b/tests/scenarios/file-github-issues/a-duplicate-gets-a-comment.yaml new file mode 100644 index 0000000..79f35bf --- /dev/null +++ b/tests/scenarios/file-github-issues/a-duplicate-gets-a-comment.yaml @@ -0,0 +1,9 @@ +name: A candidate that matches an open issue adds evidence and files nothing +skill: file-github-issues +world: + issues: + - {number: 7, state: open, labels: [ready-for-agent]} +expect: + results: ["duplicate of #7 (commented)"] + effects: + - {op: comment, target: "issue#7"} diff --git a/tests/scenarios/file-github-issues/dry-run-files-nothing.yaml b/tests/scenarios/file-github-issues/dry-run-files-nothing.yaml new file mode 100644 index 0000000..2219309 --- /dev/null +++ b/tests/scenarios/file-github-issues/dry-run-files-nothing.yaml @@ -0,0 +1,7 @@ +name: A dry run ranks and reports, and files nothing +skill: file-github-issues +inputs: {dry_run: true} +world: {issues: [], prs: []} +expect: + results: ["would-file: Search crashes on empty query"] + effects: [] diff --git a/tests/scenarios/file-github-issues/limit-defers-the-rest.yaml b/tests/scenarios/file-github-issues/limit-defers-the-rest.yaml new file mode 100644 index 0000000..c95fa03 --- /dev/null +++ b/tests/scenarios/file-github-issues/limit-defers-the-rest.yaml @@ -0,0 +1,10 @@ +name: Past the limit the remaining candidates are deferred, unchecked +why: "Gap 3. There was no limit input, so explore could not enforce explore_batch." +skill: file-github-issues +inputs: {limit: 2} +world: {issues: [], prs: []} +expect: + results: ["filed #101", "filed #102", "deferred: over limit"] + effects: + - {op: create_issue, number: 101, labels: []} + - {op: create_issue, number: 102, labels: []} diff --git a/tests/scenarios/file-github-issues/out-of-scope-is-dropped.yaml b/tests/scenarios/file-github-issues/out-of-scope-is-dropped.yaml new file mode 100644 index 0000000..c86b3a3 --- /dev/null +++ b/tests/scenarios/file-github-issues/out-of-scope-is-dropped.yaml @@ -0,0 +1,6 @@ +name: A candidate that matches an out-of-scope decision is dropped +skill: file-github-issues +world: {issues: [], prs: []} +expect: + results: ["dropped: matches .out-of-scope/"] + effects: [] diff --git a/tests/scenarios/implement-github-issue/a-blocker-is-escalated.yaml b/tests/scenarios/implement-github-issue/a-blocker-is-escalated.yaml new file mode 100644 index 0000000..a9f32c2 --- /dev/null +++ b/tests/scenarios/implement-github-issue/a-blocker-is-escalated.yaml @@ -0,0 +1,12 @@ +name: A blocker the agent cannot resolve is escalated +skill: implement-github-issue +target: issue#12 +world: + issues: + - {number: 12, state: open, labels: [ready-for-agent]} +expect: + result: "#12 escalated: needs a design decision" + effects: + - {op: comment, target: "issue#12", marker: agent:escalate} + - {op: label_remove, target: "issue#12", label: ready-for-agent} + - {op: label_add, target: "issue#12", label: agent:stuck} diff --git a/tests/scenarios/implement-github-issue/a-failed-run-leaves-the-labels.yaml b/tests/scenarios/implement-github-issue/a-failed-run-leaves-the-labels.yaml new file mode 100644 index 0000000..3fb758f --- /dev/null +++ b/tests/scenarios/implement-github-issue/a-failed-run-leaves-the-labels.yaml @@ -0,0 +1,10 @@ +name: A failed run leaves ready-for-agent in place so it is retried +skill: implement-github-issue +target: issue#12 +world: + issues: + - {number: 12, state: open, labels: [ready-for-agent]} +expect: + result: "#12 failed: build did not pass" + effects: + - {op: comment, target: "issue#12", marker: agent:failure} diff --git a/tests/scenarios/implement-github-issue/already-has-a-state-is-skipped.yaml b/tests/scenarios/implement-github-issue/already-has-a-state-is-skipped.yaml new file mode 100644 index 0000000..47eff35 --- /dev/null +++ b/tests/scenarios/implement-github-issue/already-has-a-state-is-skipped.yaml @@ -0,0 +1,9 @@ +name: An issue that is not ready-for-agent is skipped +skill: implement-github-issue +target: issue#12 +world: + issues: + - {number: 12, state: open, labels: [agent:pr-open]} +expect: + result: "#12 skipped: not ready-for-agent" + effects: [] diff --git a/tests/scenarios/implement-github-issue/existing-merged-pr-closes-the-issue.yaml b/tests/scenarios/implement-github-issue/existing-merged-pr-closes-the-issue.yaml new file mode 100644 index 0000000..712ba97 --- /dev/null +++ b/tests/scenarios/implement-github-issue/existing-merged-pr-closes-the-issue.yaml @@ -0,0 +1,13 @@ +name: An issue whose PR already merged is closed, not re-implemented +skill: implement-github-issue +target: issue#12 +world: + issues: + - {number: 12, state: open, labels: [ready-for-agent]} + prs: + - {number: 40, state: merged, labels: [], head: h1, closes: [12]} +expect: + result: "#12 already merged in #40" + effects: + - {op: label_remove, target: "issue#12", label: ready-for-agent} + - {op: close, target: "issue#12"} diff --git a/tests/scenarios/implement-github-issue/existing-open-pr-is-not-duplicated.yaml b/tests/scenarios/implement-github-issue/existing-open-pr-is-not-duplicated.yaml new file mode 100644 index 0000000..d0c19ed --- /dev/null +++ b/tests/scenarios/implement-github-issue/existing-open-pr-is-not-duplicated.yaml @@ -0,0 +1,14 @@ +name: An open PR for the issue stops a second one +skill: implement-github-issue +target: issue#12 +world: + issues: + - {number: 12, state: open, labels: [ready-for-agent]} + prs: + - {number: 40, state: open, labels: [], head: h1, closes: [12], marker: "agent:implements"} +expect: + result: "#12 existing PR #40" + effects: + - {op: label_remove, target: "issue#12", label: ready-for-agent} + - {op: label_add, target: "issue#12", label: agent:pr-open} + - {op: label_add, target: "pr#40", label: agent:needs-work} diff --git a/tests/scenarios/implement-github-issue/merged-pr-on-a-keep-open-issue-is-ignored.yaml b/tests/scenarios/implement-github-issue/merged-pr-on-a-keep-open-issue-is-ignored.yaml new file mode 100644 index 0000000..1786d7e --- /dev/null +++ b/tests/scenarios/implement-github-issue/merged-pr-on-a-keep-open-issue-is-ignored.yaml @@ -0,0 +1,14 @@ +name: On a keep-open issue a merged PR does not mean the work shipped +skill: implement-github-issue +target: issue#12 +world: + issues: + - {number: 12, state: open, keepOpen: true, labels: [ready-for-agent]} + prs: + - {number: 40, state: merged, labels: [], head: h1, closes: []} +expect: + result: "#12 PR #41 opened" + effects: + - {op: create_pr, number: 41, labels: [agent:needs-work], closes: []} + - {op: label_remove, target: "issue#12", label: ready-for-agent} + - {op: label_add, target: "issue#12", label: agent:pr-open} diff --git a/tests/scenarios/implement-github-issue/missing-information-bails-out.yaml b/tests/scenarios/implement-github-issue/missing-information-bails-out.yaml new file mode 100644 index 0000000..68402ba --- /dev/null +++ b/tests/scenarios/implement-github-issue/missing-information-bails-out.yaml @@ -0,0 +1,12 @@ +name: An issue that is not actually ready is bailed out to needs-info +skill: implement-github-issue +target: issue#12 +world: + issues: + - {number: 12, state: open, labels: [ready-for-agent]} +expect: + result: "#12 needs-info" + effects: + - {op: comment, target: "issue#12"} + - {op: label_remove, target: "issue#12", label: ready-for-agent} + - {op: label_add, target: "issue#12", label: needs-info} diff --git a/tests/scenarios/implement-github-issue/only-closed-prs-go-to-a-person.yaml b/tests/scenarios/implement-github-issue/only-closed-prs-go-to-a-person.yaml new file mode 100644 index 0000000..2cc190b --- /dev/null +++ b/tests/scenarios/implement-github-issue/only-closed-prs-go-to-a-person.yaml @@ -0,0 +1,15 @@ +name: An issue whose PRs were all closed unmerged is not retried +why: "Sending it back to an agent is how the same fix gets re-attempted in a loop." +skill: implement-github-issue +target: issue#12 +world: + issues: + - {number: 12, state: open, labels: [ready-for-agent]} + prs: + - {number: 40, state: closed, labels: [], head: h1, closes: [12]} +expect: + result: "#12 ready-for-human" + effects: + - {op: label_remove, target: "issue#12", label: ready-for-agent} + - {op: label_add, target: "issue#12", label: ready-for-human} + - {op: comment, target: "issue#12"} diff --git a/tests/scenarios/reconcile-github-issues/a-trusted-outcome-picks-the-winner.yaml b/tests/scenarios/reconcile-github-issues/a-trusted-outcome-picks-the-winner.yaml new file mode 100644 index 0000000..9c3b20b --- /dev/null +++ b/tests/scenarios/reconcile-github-issues/a-trusted-outcome-picks-the-winner.yaml @@ -0,0 +1,13 @@ +name: When a trusted outcome names the state, the other labels are removed +skill: reconcile-github-issues +world: + issues: + - number: 12 + state: open + labels: [needs-info, ready-for-human] + comments: + - {author: agent, trusted: true, marker: "agent:outcome"} +expect: + results: ["#12 repaired: needs-info and ready-for-human to needs-info"] + effects: + - {op: label_remove, target: "issue#12", label: ready-for-human} diff --git a/tests/scenarios/reconcile-github-issues/abandoned-pr-goes-to-a-person.yaml b/tests/scenarios/reconcile-github-issues/abandoned-pr-goes-to-a-person.yaml new file mode 100644 index 0000000..97d1c6d --- /dev/null +++ b/tests/scenarios/reconcile-github-issues/abandoned-pr-goes-to-a-person.yaml @@ -0,0 +1,13 @@ +name: An issue whose PRs were all closed unmerged goes to a person +skill: reconcile-github-issues +world: + issues: + - {number: 12, state: open, labels: [agent:pr-open]} + prs: + - {number: 40, state: closed, labels: [], head: h1, closes: [12]} +expect: + results: ["#12 ready-for-human: PR #40 closed without merging"] + effects: + - {op: label_remove, target: "issue#12", label: agent:pr-open} + - {op: label_add, target: "issue#12", label: ready-for-human} + - {op: comment, target: "issue#12"} diff --git a/tests/scenarios/reconcile-github-issues/conflicting-labels-fall-back-to-a-person.yaml b/tests/scenarios/reconcile-github-issues/conflicting-labels-fall-back-to-a-person.yaml new file mode 100644 index 0000000..bb588eb --- /dev/null +++ b/tests/scenarios/reconcile-github-issues/conflicting-labels-fall-back-to-a-person.yaml @@ -0,0 +1,10 @@ +name: Conflicting state labels with no clear winner fall back to ready-for-human +skill: reconcile-github-issues +world: + issues: + - {number: 12, state: open, labels: [ready-for-agent, ready-for-human]} +expect: + results: ["#12 ready-for-human: conflicting state labels"] + effects: + - {op: label_remove, target: "issue#12", label: ready-for-agent} + - {op: comment, target: "issue#12"} diff --git a/tests/scenarios/reconcile-github-issues/dry-run-writes-nothing.yaml b/tests/scenarios/reconcile-github-issues/dry-run-writes-nothing.yaml new file mode 100644 index 0000000..470413a --- /dev/null +++ b/tests/scenarios/reconcile-github-issues/dry-run-writes-nothing.yaml @@ -0,0 +1,11 @@ +name: A dry run reports what would change and writes nothing +skill: reconcile-github-issues +inputs: {dry_run: true} +world: + issues: + - {number: 12, state: open, labels: [agent:pr-open]} + prs: + - {number: 40, state: merged, labels: [], head: h1, closes: [12]} +expect: + results: ["would: #12 closed: implemented in #40"] + effects: [] diff --git a/tests/scenarios/reconcile-github-issues/keep-open-stays-open.yaml b/tests/scenarios/reconcile-github-issues/keep-open-stays-open.yaml new file mode 100644 index 0000000..6005147 --- /dev/null +++ b/tests/scenarios/reconcile-github-issues/keep-open-stays-open.yaml @@ -0,0 +1,13 @@ +name: A keep-open issue is never closed +skill: reconcile-github-issues +world: + issues: + - {number: 12, state: open, keepOpen: true, labels: [agent:pr-open]} + prs: + - {number: 40, state: merged, labels: [], head: h1, closes: [12]} +expect: + results: ["#12 kept open: merged in #40"] + effects: + - {op: label_remove, target: "issue#12", label: agent:pr-open} + - {op: comment, target: "issue#12"} + - {op: label_add, target: "issue#12", label: ready-for-human} diff --git a/tests/scenarios/reconcile-github-issues/live-lease-is-left-alone.yaml b/tests/scenarios/reconcile-github-issues/live-lease-is-left-alone.yaml new file mode 100644 index 0000000..f3078eb --- /dev/null +++ b/tests/scenarios/reconcile-github-issues/live-lease-is-left-alone.yaml @@ -0,0 +1,8 @@ +name: An issue held by a live lease is not repaired +skill: reconcile-github-issues +world: + issues: + - {number: 14, state: open, labels: [needs-info, ready-for-agent, "agent:triage"], lease: live} +expect: + results: ["nothing to do"] + effects: [] diff --git a/tests/scenarios/reconcile-github-issues/merged-pr-closes-the-issue.yaml b/tests/scenarios/reconcile-github-issues/merged-pr-closes-the-issue.yaml new file mode 100644 index 0000000..cdbe504 --- /dev/null +++ b/tests/scenarios/reconcile-github-issues/merged-pr-closes-the-issue.yaml @@ -0,0 +1,12 @@ +name: An issue whose PR merged is closed +skill: reconcile-github-issues +world: + issues: + - {number: 12, state: open, labels: [agent:pr-open]} + prs: + - {number: 40, state: merged, labels: [], head: h1, closes: [12]} +expect: + results: ["#12 closed: implemented in #40"] + effects: + - {op: label_remove, target: "issue#12", label: agent:pr-open} + - {op: close, target: "issue#12"} diff --git a/tests/scenarios/reconcile-github-issues/stale-lease-is-cleared.yaml b/tests/scenarios/reconcile-github-issues/stale-lease-is-cleared.yaml new file mode 100644 index 0000000..78b3efa --- /dev/null +++ b/tests/scenarios/reconcile-github-issues/stale-lease-is-cleared.yaml @@ -0,0 +1,10 @@ +name: A lease label with no live lease is removed +why: "Gap 5. Eligibility used to do this as a side effect. Now only the sweeper does." +skill: reconcile-github-issues +world: + issues: + - {number: 13, state: open, labels: [needs-triage, "agent:in-progress"], lease: stale} +expect: + results: ["#13 stale lease cleared"] + effects: + - {op: label_remove, target: "issue#13", label: "agent:in-progress"} diff --git a/tests/scenarios/review-github-pr/a-fork-is-escalated.yaml b/tests/scenarios/review-github-pr/a-fork-is-escalated.yaml new file mode 100644 index 0000000..b45c43a --- /dev/null +++ b/tests/scenarios/review-github-pr/a-fork-is-escalated.yaml @@ -0,0 +1,15 @@ +name: A fork PR cannot be pushed to, so it is escalated +skill: review-github-pr +target: pr#50 +world: + issues: + - {number: 12, state: open, labels: [agent:pr-open]} + prs: + - {number: 50, state: open, labels: [agent:ready-for-review], head: h1, readySha: h1, base: main, mergeState: CLEAN, checks: [pass], closes: [12], fork: true, files: []} +expect: + result: "#50 escalated: fork PR" + effects: + - {op: comment, target: "pr#50", marker: agent:escalate} + - {op: label_remove, target: "pr#50", label: agent:ready-for-review} + - {op: label_add, target: "pr#50", label: agent:stuck} + - {op: comment, target: "issue#12"} diff --git a/tests/scenarios/review-github-pr/a-rejected-merge-is-a-failed-run-not-a-retry.yaml b/tests/scenarios/review-github-pr/a-rejected-merge-is-a-failed-run-not-a-retry.yaml new file mode 100644 index 0000000..568f76c --- /dev/null +++ b/tests/scenarios/review-github-pr/a-rejected-merge-is-a-failed-run-not-a-retry.yaml @@ -0,0 +1,10 @@ +name: A merge that GitHub rejects fails the run and is not retried +skill: review-github-pr +target: pr#50 +world: + prs: + - {number: 50, state: open, labels: [agent:ready-for-review], head: h1, readySha: h1, base: main, mergeState: CLEAN, checks: [pass], files: []} +expect: + result: "#50 failed: merge rejected" + effects: + - {op: comment, target: "pr#50", marker: agent:failure} diff --git a/tests/scenarios/review-github-pr/base-moved-is-deferred.yaml b/tests/scenarios/review-github-pr/base-moved-is-deferred.yaml new file mode 100644 index 0000000..63984ef --- /dev/null +++ b/tests/scenarios/review-github-pr/base-moved-is-deferred.yaml @@ -0,0 +1,12 @@ +name: A branch behind its base is deferred, without spending a review round +skill: review-github-pr +target: pr#50 +world: + prs: + - {number: 50, state: open, labels: [agent:ready-for-review], head: h1, readySha: h1, base: main, mergeState: BEHIND, checks: [pass], closes: [12], files: [src/a.js]} +expect: + result: "#50 deferred: base moved" + effects: + - {op: comment, target: "pr#50", marker: agent:deferred} + - {op: label_remove, target: "pr#50", label: agent:ready-for-review} + - {op: label_add, target: "pr#50", label: agent:needs-work} diff --git a/tests/scenarios/review-github-pr/findings-are-sent-back.yaml b/tests/scenarios/review-github-pr/findings-are-sent-back.yaml new file mode 100644 index 0000000..6d836aa --- /dev/null +++ b/tests/scenarios/review-github-pr/findings-are-sent-back.yaml @@ -0,0 +1,12 @@ +name: Open findings send the PR back to shepherding with a review hand-off +skill: review-github-pr +target: pr#50 +world: + prs: + - {number: 50, state: open, labels: [agent:ready-for-review], head: h1, readySha: h1, base: main, mergeState: CLEAN, checks: [pass], files: []} +expect: + result: "#50 sent back: 2 open findings" + effects: + - {op: comment, target: "pr#50", marker: agent:handoff} + - {op: label_remove, target: "pr#50", label: agent:ready-for-review} + - {op: label_add, target: "pr#50", label: agent:needs-work} diff --git a/tests/scenarios/review-github-pr/head-moved-after-the-handoff.yaml b/tests/scenarios/review-github-pr/head-moved-after-the-handoff.yaml new file mode 100644 index 0000000..e6607c9 --- /dev/null +++ b/tests/scenarios/review-github-pr/head-moved-after-the-handoff.yaml @@ -0,0 +1,12 @@ +name: A head that moved after the hand-off is sent back with a result line +why: "Gap 2. Step 4 used to swap the label and stop with no result." +skill: review-github-pr +target: pr#50 +world: + prs: + - {number: 50, state: open, labels: [agent:ready-for-review], head: h2, readySha: h1, base: main, mergeState: CLEAN, checks: [pass], files: []} +expect: + result: "#50 not ready: head moved after the handoff" + effects: + - {op: label_remove, target: "pr#50", label: agent:ready-for-review} + - {op: label_add, target: "pr#50", label: agent:needs-work} diff --git a/tests/scenarios/review-github-pr/merge-closes-the-linked-issue.yaml b/tests/scenarios/review-github-pr/merge-closes-the-linked-issue.yaml new file mode 100644 index 0000000..9617400 --- /dev/null +++ b/tests/scenarios/review-github-pr/merge-closes-the-linked-issue.yaml @@ -0,0 +1,16 @@ +name: A clean review merges the pinned head and closes the issue +skill: review-github-pr +target: pr#50 +world: + issues: + - {number: 12, state: open, labels: [agent:pr-open]} + prs: + - {number: 50, state: open, labels: [agent:ready-for-review], head: h1, readySha: h1, base: main, mergeState: CLEAN, checks: [pass], closes: [12], files: [src/a.js]} +expect: + result: "#50 merged" + effects: + - {op: merge, target: "pr#50", sha: h1} + - {op: delete_branch, target: "pr#50"} + - {op: comment, target: "pr#50", marker: agent:handoff} + - {op: label_remove, target: "issue#12", label: agent:pr-open} + - {op: close, target: "issue#12"} diff --git a/tests/scenarios/review-github-pr/merge-leaves-a-keep-open-issue-open.yaml b/tests/scenarios/review-github-pr/merge-leaves-a-keep-open-issue-open.yaml new file mode 100644 index 0000000..afb12ab --- /dev/null +++ b/tests/scenarios/review-github-pr/merge-leaves-a-keep-open-issue-open.yaml @@ -0,0 +1,18 @@ +name: A keep-open issue is not closed by the merge +why: "Closing it would let an agent or GitHub finish an issue that is meant to stay open. After a merge it goes to a person." +skill: review-github-pr +target: pr#50 +world: + issues: + - {number: 12, state: open, keepOpen: true, labels: [agent:pr-open]} + prs: + - {number: 50, state: open, labels: [agent:ready-for-review], head: h1, readySha: h1, base: main, mergeState: CLEAN, checks: [pass], closes: [12], files: [src/a.js]} +expect: + result: "#50 merged" + effects: + - {op: merge, target: "pr#50", sha: h1} + - {op: delete_branch, target: "pr#50"} + - {op: comment, target: "pr#50", marker: agent:handoff} + - {op: label_remove, target: "issue#12", label: agent:pr-open} + - {op: comment, target: "issue#12"} + - {op: label_add, target: "issue#12", label: ready-for-human} diff --git a/tests/scenarios/review-github-pr/not-ready-for-review-is-skipped.yaml b/tests/scenarios/review-github-pr/not-ready-for-review-is-skipped.yaml new file mode 100644 index 0000000..f314129 --- /dev/null +++ b/tests/scenarios/review-github-pr/not-ready-for-review-is-skipped.yaml @@ -0,0 +1,9 @@ +name: A PR that is not labelled ready-for-review is skipped +skill: review-github-pr +target: pr#50 +world: + prs: + - {number: 50, state: open, labels: [agent:human-merge], head: h1, base: main, mergeState: CLEAN, checks: [pass], files: []} +expect: + result: "#50 skipped: not agent:ready-for-review" + effects: [] diff --git a/tests/scenarios/review-github-pr/protected-path-goes-to-a-person.yaml b/tests/scenarios/review-github-pr/protected-path-goes-to-a-person.yaml new file mode 100644 index 0000000..7e34fcc --- /dev/null +++ b/tests/scenarios/review-github-pr/protected-path-goes-to-a-person.yaml @@ -0,0 +1,13 @@ +name: A PR that changes a protected path is never merged by the agent +skill: review-github-pr +target: pr#50 +inputs: {human_merge_paths: [".github/"]} +world: + prs: + - {number: 50, state: open, labels: [agent:ready-for-review], head: h1, readySha: h1, base: main, mergeState: CLEAN, checks: [pass], closes: [12], files: [.github/workflows/ci.yml]} +expect: + result: "#50 handed to a human: protected-paths" + effects: + - {op: comment, target: "pr#50", marker: agent:human-merge} + - {op: label_remove, target: "pr#50", label: agent:ready-for-review} + - {op: label_add, target: "pr#50", label: agent:human-merge} diff --git a/tests/scenarios/shepherd-github-pr/behind-is-updated-not-handed-off.yaml b/tests/scenarios/shepherd-github-pr/behind-is-updated-not-handed-off.yaml new file mode 100644 index 0000000..57ec2db --- /dev/null +++ b/tests/scenarios/shepherd-github-pr/behind-is-updated-not-handed-off.yaml @@ -0,0 +1,12 @@ +name: A branch that is behind is updated, and is not yet handed off +why: "An update is not a fix attempt, and the PR stays agent:needs-work." +skill: shepherd-github-pr +target: pr#40 +world: + prs: + - {number: 40, state: open, labels: [agent:needs-work], head: h1, base: main, mergeState: BEHIND, checks: [pass], openFindings: 0, closes: [12]} +expect: + result: "#40 branch updated" + effects: + - {op: push, target: "pr#40", sha: h2} + - {op: comment, target: "pr#40", marker: agent:branch-update} diff --git a/tests/scenarios/shepherd-github-pr/conflict-without-checks.yaml b/tests/scenarios/shepherd-github-pr/conflict-without-checks.yaml new file mode 100644 index 0000000..b2b6199 --- /dev/null +++ b/tests/scenarios/shepherd-github-pr/conflict-without-checks.yaml @@ -0,0 +1,12 @@ +name: A conflicted PR that runs no checks is still worked on +why: "Gap 9. Eligibility used to admit only failing or green checks, so a DIRTY branch with no checks was skipped as no-checks-yet and its conflict was never fixed." +skill: shepherd-github-pr +target: pr#40 +world: + prs: + - {number: 40, state: open, labels: [agent:needs-work], head: h1, base: main, mergeState: DIRTY, checks: [], openFindings: 0, closes: [12]} +expect: + result: "#40 fix pushed" + effects: + - {op: push, target: "pr#40", sha: h2} + - {op: comment, target: "pr#40", marker: agent:attempt} diff --git a/tests/scenarios/shepherd-github-pr/escalates-at-the-attempt-limit.yaml b/tests/scenarios/shepherd-github-pr/escalates-at-the-attempt-limit.yaml new file mode 100644 index 0000000..cdf541c --- /dev/null +++ b/tests/scenarios/shepherd-github-pr/escalates-at-the-attempt-limit.yaml @@ -0,0 +1,15 @@ +name: A PR that has used its fix attempts is escalated, not tried again +skill: shepherd-github-pr +target: pr#40 +world: + issues: + - {number: 12, state: open, labels: [agent:pr-open]} + prs: + - {number: 40, state: open, labels: [agent:needs-work], head: h1, base: main, mergeState: CLEAN, checks: [fail], openFindings: 0, closes: [12], shepherdAttempts: 3} +expect: + result: "#40 escalated: 3 fix attempts used" + effects: + - {op: comment, target: "pr#40", marker: agent:escalate} + - {op: label_remove, target: "pr#40", label: agent:needs-work} + - {op: label_add, target: "pr#40", label: agent:stuck} + - {op: comment, target: "issue#12"} diff --git a/tests/scenarios/shepherd-github-pr/failing-checks-are-fixed-while-the-lane-is-held.yaml b/tests/scenarios/shepherd-github-pr/failing-checks-are-fixed-while-the-lane-is-held.yaml new file mode 100644 index 0000000..099a14b --- /dev/null +++ b/tests/scenarios/shepherd-github-pr/failing-checks-are-fixed-while-the-lane-is-held.yaml @@ -0,0 +1,13 @@ +name: A failing PR is fixed even when another PR holds the lane +why: "The lane only matters for a hand-off or a branch update, not for a fix." +skill: shepherd-github-pr +target: pr#40 +world: + prs: + - {number: 40, state: open, labels: [agent:needs-work], head: h1, base: main, mergeState: CLEAN, checks: [fail], openFindings: 0, closes: [12]} + - {number: 41, state: open, labels: [agent:ready-for-review], head: h9, base: main, mergeState: CLEAN, checks: [pass]} +expect: + result: "#40 fix pushed" + effects: + - {op: push, target: "pr#40", sha: h2} + - {op: comment, target: "pr#40", marker: agent:attempt} diff --git a/tests/scenarios/shepherd-github-pr/green-and-lane-free-hands-off.yaml b/tests/scenarios/shepherd-github-pr/green-and-lane-free-hands-off.yaml new file mode 100644 index 0000000..e86da94 --- /dev/null +++ b/tests/scenarios/shepherd-github-pr/green-and-lane-free-hands-off.yaml @@ -0,0 +1,12 @@ +name: Green, current, no findings and a free lane is handed off +skill: shepherd-github-pr +target: pr#40 +world: + prs: + - {number: 40, state: open, labels: [agent:needs-work], head: h1, base: main, mergeState: CLEAN, checks: [pass], openFindings: 0, closes: [12]} +expect: + result: "#40 ready-for-review" + effects: + - {op: comment, target: "pr#40", marker: agent:handoff} + - {op: label_remove, target: "pr#40", label: agent:needs-work} + - {op: label_add, target: "pr#40", label: agent:ready-for-review} diff --git a/tests/scenarios/shepherd-github-pr/lane-held-is-skipped-without-a-trace.yaml b/tests/scenarios/shepherd-github-pr/lane-held-is-skipped-without-a-trace.yaml new file mode 100644 index 0000000..a360c49 --- /dev/null +++ b/tests/scenarios/shepherd-github-pr/lane-held-is-skipped-without-a-trace.yaml @@ -0,0 +1,11 @@ +name: A PR that can only wait on the merge lane is skipped before it is claimed +why: "Gap 9. It used to be claimed and released on every run, leaving a lease comment each time. Skipped means no effects at all." +skill: shepherd-github-pr +target: pr#40 +world: + prs: + - {number: 40, state: open, labels: [agent:needs-work], head: h1, base: main, mergeState: CLEAN, checks: [pass], openFindings: 0, closes: [12]} + - {number: 41, state: open, labels: [agent:ready-for-review], head: h9, base: main, mergeState: CLEAN, checks: [pass]} +expect: + result: "#40 skipped: merge lane held by #41" + effects: [] diff --git a/tests/scenarios/shepherd-github-pr/lane-taken-after-the-claim.yaml b/tests/scenarios/shepherd-github-pr/lane-taken-after-the-claim.yaml new file mode 100644 index 0000000..05606eb --- /dev/null +++ b/tests/scenarios/shepherd-github-pr/lane-taken-after-the-claim.yaml @@ -0,0 +1,14 @@ +name: A lane taken between the read and the claim is a noop +why: "Gap 2. The PR was eligible when read, so it was claimed. When another run took the lane it is released unchanged. The claim's lease comment is the only trace." +skill: shepherd-github-pr +target: pr#40 +eligibility_world: + prs: + - {number: 40, state: open, labels: [agent:needs-work], head: h1, base: main, mergeState: CLEAN, checks: [pass], openFindings: 0, closes: [12]} +world: + prs: + - {number: 40, state: open, labels: [agent:needs-work], head: h1, base: main, mergeState: CLEAN, checks: [pass], openFindings: 0, closes: [12]} + - {number: 41, state: open, labels: [agent:ready-for-review], head: h9, base: main, mergeState: CLEAN, checks: [pass]} +expect: + result: "#40 noop: merge lane held by #41" + effects: [] diff --git a/tests/scenarios/shepherd-github-pr/pending-checks-are-skipped.yaml b/tests/scenarios/shepherd-github-pr/pending-checks-are-skipped.yaml new file mode 100644 index 0000000..fd83435 --- /dev/null +++ b/tests/scenarios/shepherd-github-pr/pending-checks-are-skipped.yaml @@ -0,0 +1,10 @@ +name: Running checks mean skip, not wait +why: "The skill never polls CI." +skill: shepherd-github-pr +target: pr#40 +world: + prs: + - {number: 40, state: open, labels: [agent:needs-work], head: h1, base: main, mergeState: CLEAN, checks: [pending], openFindings: 0, closes: [12]} +expect: + result: "#40 skipped: checks pending" + effects: [] diff --git a/tests/scenarios/shepherd-github-pr/unreadable-failure-is-a-failed-run.yaml b/tests/scenarios/shepherd-github-pr/unreadable-failure-is-a-failed-run.yaml new file mode 100644 index 0000000..a8ac961 --- /dev/null +++ b/tests/scenarios/shepherd-github-pr/unreadable-failure-is-a-failed-run.yaml @@ -0,0 +1,10 @@ +name: Failing output that cannot be read is a failed run, not an attempt +skill: shepherd-github-pr +target: pr#40 +world: + prs: + - {number: 40, state: open, labels: [agent:needs-work], head: h1, base: main, mergeState: CLEAN, checks: [fail], openFindings: 0} +expect: + result: "#40 failed: cannot read failing output" + effects: + - {op: comment, target: "pr#40", marker: agent:failure} diff --git a/tests/scenarios/triage-github-issue/a-failed-pass-leaves-the-state.yaml b/tests/scenarios/triage-github-issue/a-failed-pass-leaves-the-state.yaml new file mode 100644 index 0000000..2d6c1bf --- /dev/null +++ b/tests/scenarios/triage-github-issue/a-failed-pass-leaves-the-state.yaml @@ -0,0 +1,10 @@ +name: A failed pass leaves the state labels as they were +skill: triage-github-issue +target: issue#12 +world: + issues: + - {number: 12, state: open, labels: [needs-triage]} +expect: + result: "#12 failed: could not read the attachment" + effects: + - {op: comment, target: "issue#12", marker: agent:failure} diff --git a/tests/scenarios/triage-github-issue/a-human-answer-supersedes-a-prior-outcome.yaml b/tests/scenarios/triage-github-issue/a-human-answer-supersedes-a-prior-outcome.yaml new file mode 100644 index 0000000..0244758 --- /dev/null +++ b/tests/scenarios/triage-github-issue/a-human-answer-supersedes-a-prior-outcome.yaml @@ -0,0 +1,18 @@ +name: A human answer after the prior outcome means deciding again +skill: triage-github-issue +target: issue#12 +world: + issues: + - number: 12 + state: open + labels: [needs-triage] + comments: + - {author: agent, trusted: true, marker: "agent:outcome"} + - {author: human} +expect: + result: "#12 ready-for-agent" + calls: [unattended-triage] + effects: + - {op: label_add, target: "issue#12", label: ready-for-agent} + - {op: label_remove, target: "issue#12", label: needs-triage} + - {op: comment, target: "issue#12"} diff --git a/tests/scenarios/triage-github-issue/a-standing-prior-outcome-is-reused.yaml b/tests/scenarios/triage-github-issue/a-standing-prior-outcome-is-reused.yaml new file mode 100644 index 0000000..85e3353 --- /dev/null +++ b/tests/scenarios/triage-github-issue/a-standing-prior-outcome-is-reused.yaml @@ -0,0 +1,18 @@ +name: A trusted prior outcome that still stands is reused without deciding again +why: "Gap 8. The check used to run after unattended-triage, so a pass that had already decided paid for the decision twice." +skill: triage-github-issue +target: issue#12 +world: + issues: + - number: 12 + state: open + labels: [needs-triage] + comments: + - {author: agent, trusted: true, marker: "agent:outcome"} +expect: + result: "#12 ready-for-agent" + calls: [] + effects: + - {op: label_add, target: "issue#12", label: ready-for-agent} + - {op: label_remove, target: "issue#12", label: needs-triage} + - {op: comment, target: "issue#12"} diff --git a/tests/scenarios/triage-github-issue/already-triaged-is-skipped.yaml b/tests/scenarios/triage-github-issue/already-triaged-is-skipped.yaml new file mode 100644 index 0000000..dfec5c2 --- /dev/null +++ b/tests/scenarios/triage-github-issue/already-triaged-is-skipped.yaml @@ -0,0 +1,9 @@ +name: An issue that already has a state label is not triaged again +skill: triage-github-issue +target: issue#12 +world: + issues: + - {number: 12, state: open, labels: [ready-for-agent]} +expect: + result: "#12 skipped: already has a state label" + effects: [] diff --git a/tests/scenarios/triage-github-issue/duplicate-is-closed.yaml b/tests/scenarios/triage-github-issue/duplicate-is-closed.yaml new file mode 100644 index 0000000..e3613a8 --- /dev/null +++ b/tests/scenarios/triage-github-issue/duplicate-is-closed.yaml @@ -0,0 +1,12 @@ +name: A duplicate is closed and links to the original +skill: triage-github-issue +target: issue#12 +world: + issues: + - {number: 7, state: open, labels: [ready-for-agent]} + - {number: 12, state: open, labels: [needs-triage]} +expect: + result: "#12 duplicate of #7" + effects: + - {op: comment, target: "issue#12"} + - {op: close, target: "issue#12"} diff --git a/tests/scenarios/triage-github-issue/open-questions-mean-needs-info.yaml b/tests/scenarios/triage-github-issue/open-questions-mean-needs-info.yaml new file mode 100644 index 0000000..8a0e5f1 --- /dev/null +++ b/tests/scenarios/triage-github-issue/open-questions-mean-needs-info.yaml @@ -0,0 +1,13 @@ +name: A missing required section means needs-info, never ready-for-agent +skill: triage-github-issue +target: issue#12 +world: + issues: + - {number: 12, state: open, labels: [needs-triage]} +expect: + result: "#12 needs-info" + calls: [unattended-triage] + effects: + - {op: label_add, target: "issue#12", label: needs-info} + - {op: label_remove, target: "issue#12", label: needs-triage} + - {op: comment, target: "issue#12"} diff --git a/tests/scenarios/triage-github-issue/repeated-failures-escalate.yaml b/tests/scenarios/triage-github-issue/repeated-failures-escalate.yaml new file mode 100644 index 0000000..15e2806 --- /dev/null +++ b/tests/scenarios/triage-github-issue/repeated-failures-escalate.yaml @@ -0,0 +1,11 @@ +name: An issue that keeps failing triage is escalated +skill: triage-github-issue +target: issue#12 +world: + issues: + - {number: 12, state: open, labels: [needs-triage]} +expect: + result: "#12 escalated: failed 3 times" + effects: + - {op: comment, target: "issue#12", marker: agent:escalate} + - {op: label_add, target: "issue#12", label: agent:stuck} diff --git a/tests/scenarios/triage-github-issue/wontfix-is-ready-for-human-not-closed.yaml b/tests/scenarios/triage-github-issue/wontfix-is-ready-for-human-not-closed.yaml new file mode 100644 index 0000000..f921837 --- /dev/null +++ b/tests/scenarios/triage-github-issue/wontfix-is-ready-for-human-not-closed.yaml @@ -0,0 +1,13 @@ +name: A wontfix recommendation goes to a person and the issue stays open +skill: triage-github-issue +target: issue#12 +world: + issues: + - {number: 12, state: open, labels: [needs-triage]} +expect: + result: "#12 ready-for-human" + calls: [unattended-triage] + effects: + - {op: label_add, target: "issue#12", label: ready-for-human} + - {op: label_remove, target: "issue#12", label: needs-triage} + - {op: comment, target: "issue#12"}