Skip to content

feat(ops): report deterministic remediation facts on the incident packet - #351

Merged
i-xtsu-sixyou-ken-mei merged 7 commits into
mainfrom
feat/ops-autonomy-policy
Sep 2, 2026
Merged

feat(ops): report deterministic remediation facts on the incident packet#351
i-xtsu-sixyou-ken-mei merged 7 commits into
mainfrom
feat/ops-autonomy-policy

Conversation

@i-xtsu-sixyou-ken-mei

@i-xtsu-sixyou-ken-mei i-xtsu-sixyou-ken-mei commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Intent

Give an agent a deterministic, machine-readable answer to "is this incident safe to act on yet?" and put it where the agent already looks: on the ops_investigate packet. Operational priority stays what it is — impact and urgency — and never doubles as permission.

The server reports facts, not a verdict. It does not grade autonomy, because it structurally cannot: whether a repair is safe depends on the kind of change it needs, and one signal can require either a one-line guard or a schema migration. Change kind is only knowable after an agent has diagnosed the root cause, so .agents/skills/ops-incident-remediation keeps that judgement and this packet block supplies the facts a skill cannot see for itself.

This PR adds no write capability. It does not add GitHub rerun, social requeue, Fly, database, customer-state, environment, or any other provider/runtime mutation.

Context

zap-pilot-ops already had strong read-side boundaries: ops_status, bounded investigation/inspection, deterministic operational priority, and one narrowly allowlisted Sentry resolve mutation. What was missing was a machine-readable statement of whether an incident is evidenced well enough to act on.

The priority engine weights customers, social and infra highest precisely because a wrong change there costs the most. An agent that took the top of ops_status.priorities as its work queue would therefore be pointed at the least safe targets first. That is the gap this closes.

Scope

  • add RemediationFacts to the ops_investigate incident packet
  • report observer trust, deep-inspection coverage, proven exposure, and terminal retry state
  • fail closed: unknown state, observer failure, a fingerprint absent from the snapshot, an unavailable or not-found inspection, any unresolved evidence gap, and any non-zero aumAtRiskUsd all become blockers
  • treat no-inspector as a caveat rather than a blocker, and forbid calling such an incident production-verified
  • keep directMutationAllowed unconditionally false
  • extract the shared evidenceNumber helper that prioritize.ts and the policy both needed
  • document the boundary in apps/control-center/MCP.md
  • route .agents/skills/ops-incident-remediation through remediation.blockers and repair its two stale MCP.md:<line> citations
  • add unit and integration regression coverage

Out of scope

  • no GitHub Actions rerun executor, no social requeue/retry executor
  • no new Sentry write behavior and no weakening of the existing resolve gate
  • no Fly scaling/restart/machine mutation, no database or migration execution
  • no environment, secret, or customer service-tier mutation
  • no wallet, transaction, investment, portfolio, or financial-semantic mutation
  • no auto-merge or auto-deploy behavior, no growth/product experiment executor
  • no change to the operational priority formula

Product contract / invariants

  • This PR does not change a documented product/architecture invariant.

The Ops MCP remains read-first and ops_resolve_sentry_issue remains the sole narrowly allowlisted provider mutation, unchanged. Operational priority remains deterministic and continues to represent impact rather than permission.

directMutationAllowed: false is an advisory contract statement, not an enforcement mechanism: the Sentry resolve tool has always enforced its own credential and shape gate and does not consult this block. The documentation says so explicitly rather than implying the packet gates writes.

Implementation

apps/control-center/src/server/services/operations/autonomy.ts owns buildRemediationFacts(). It reads the signal's status and fingerprint kind for observer trust, maps the primary SignalInspection status to a coverage value, and reads proven exposure out of signal evidence through the new shared evidence.ts helper.

investigateOperationalSignal() calls it once, after gap de-duplication, using the same snapshot, the same priority list and the same primary inspection the rest of the packet is built from. There is no second provider fan-out and no second fingerprint lookup.

Why the design changed since the first draft of this PR

The first draft added a standalone ops_assess_remediation tool that graded each signal into observe / auto-pr / approval-required from a table keyed by the signal's domain. Review found three problems that the current design fixes.

The safety check was inverted. Only github-actions, sentry and fly have deep inspectors. Every other source returns unsupportedInspection, which hard-codes an empty gap list. The old policy demoted only on evidenceGaps.length > 0, so social, product, costs, customers and posthog signals were reported as evidence-ready and reached auto-pr because nothing had been gathered. Coverage is now derived from the inspection status, and "no inspector ran" is visibly different from "inspection came back clean".

The grading axis was wrong. Domain does not determine whether a repair is safe; change kind does, and a signal cannot carry change kind. The domain table simultaneously over-authorized (a jobs signal whose real fix is a broad timeout policy change, which the skill forbids) and over-restricted (an infra signal whose real fix is a one-line config typo). The grades are gone; the skill's existing change-kind rules are the single home for that judgement.

Two supporting defects. The standalone tool fetched a snapshot and then called ops_investigate, which fetched the snapshot again — two full provider fan-outs inside the 30-second MCP function budget, with the signal and the gaps coming from different snapshots. It also re-implemented projectSignal and re-declared EvidenceGap. Folding the facts into the packet removes all of that: one snapshot, one fan-out, and no tool that nothing calls.

Acceptance criteria

  • ops_investigate returns remediation with policy version, operational priority score, observer trust, inspection coverage, exposure, terminal state, blockers and reasons.
  • directMutationAllowed is always false.
  • A high operational priority does not imply an empty blocker list.
  • A source with no deep inspector never reads as verified evidence, and its empty gap list cannot be mistaken for clean evidence.
  • unknown state, observer failure, and a fingerprint absent from the snapshot all block.
  • Any unresolved investigation evidence gap becomes a blocker.
  • Non-zero aumAtRiskUsd blocks.
  • The existing Sentry mutation contract is unchanged.
  • The incident-remediation skill consumes the block and its MCP.md citations resolve.

Contract alignment

  • Implementation matches the invariant.
  • Regression / contract tests cover it.
  • Scoped AGENTS.md remains accurate. apps/control-center/ has no scoped AGENTS.md, so the repository-root one applies and is unaffected.
  • README / runbook remains accurate. apps/control-center/README.md describes the write boundary, which is unchanged; MCP.md is updated in this PR.

Validation

Run in the branch worktree against real gates:

  • vitest run — 46 files, 311 tests pass
  • turbo run type-check — all three tsconfig projects pass
  • turbo run lint deadcode — 0 errors (only pre-existing curly warnings in investigation.ts)
  • turbo run format:check — clean
  • vitest run --coverage — exits 0
  • node scripts/lint/run-jscpd.mjs src — the two clones this PR introduced are gone; 8 clones remain, all pre-existing on main and none touching the changed files

Validation gaps

The remote MCP surface was not exercised against a live deployment; the HTTP protocol path is covered by the integration tests instead. No production provider was called during validation, so the facts block has been verified against fixtures and the real code path, not against live provider evidence.

Known unrelated failures

code-quality stays red until main is repaired. main fails dup:check on 8 pre-existing Control Center clones introduced by #350; #353 repairs them and is still open.

The count is the evidence that none of them belong to this PR. Run 33599314677 on the previous head reported 10 clones: those 8 plus 2 from the first draft's autonomy.ts. Run 33604577272 on the current head reports 8, the same set main reports. Every other job passes: quick-gates, tests, coverage, e2e, security, check-dead-env, deploy-gates.

Reviewer notes

This change was reviewed by five independent lenses (facts-model logic, integration wiring, documentation truthfulness, test adequacy, repository guardrails), each finding adversarially verified against the source. Nineteen candidate findings were raised and all nineteen were refuted. The exposure-scope note above is the sharpest surviving objection and is answered by documentation plus the skill rule rather than by code.

Please read autonomy.ts as a facts reporter, not as a ranking heuristic. The invariant is that the server states only what it can prove — whether the reading is trustworthy, how much evidence backs it, and what exposure is demonstrated — and refuses to pretend it knows whether a given repair is safe.

remediation.exposure reports only what the investigated signal itself proves. A correlated incident — an alpha-etl cron failure behind stale priority portfolios, say — carries its customer exposure in the packet's separate customerImpact, so the skill rule and MCP.md both tell an agent to read both. Attributing another signal's AUM to a job failure inside exposure would be an inference rather than a fact, which is the one thing this block must not do. Making that correlation blocking on the server is a reasonable follow-up; it is deliberately not in this PR.

no-inspector being a caveat rather than a blocker is deliberate. Making it a blocker would permanently freeze every social, product, costs and customers incident, since none of those sources has a deep inspector. Such incidents can still be repaired from repository evidence; what they cannot be called is production-verified.

A follow-up can add action-specific bounded executors, but each should carry its own server-generated precondition and verification contract rather than widening this block implicitly.

The first draft graded each signal into observe/auto-pr/approval-required
from a table keyed by the signal's domain. Two problems made that unsafe.

Evidence readiness was inverted. Only github-actions, sentry and fly have
deep inspectors; every other source returns unsupportedInspection, which
hard-codes an empty gap list. Demoting only on evidenceGaps.length meant
social, product, costs, customers and posthog signals were judged evidence
ready precisely because nothing had been gathered. Coverage now comes from
the inspection status, so "no inspector ran" is visibly different from
"inspection came back clean".

The grading axis was wrong. Whether a repair is safe depends on the kind of
change it needs, and a signal cannot carry change kind because that is only
knowable after root-cause diagnosis. The server now reports facts it can
prove -- observer trust, inspection coverage, exposure, terminal state and
blockers -- and leaves the change-kind judgement to the incident-remediation
skill that already owns it.

Folding the facts into the incident packet also removes the standalone tool's
double provider fan-out, its duplicate fingerprint lookup, and its re-declared
EvidenceGap type. evidenceNumber moves to a shared module rather than being
copied between the priority engine and the policy.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
MCP.md now describes the facts block on the investigation packet rather than
a standalone assessment tool, states that no-inspector is a caveat instead of
a blocker, and says plainly that exposure covers only what the investigated
signal proves so an agent also reads the packet's correlated customerImpact.

The incident-remediation skill gains the rule that consumes it and loses two
line-number citations into MCP.md that this change would have invalidated.
Section names replace them, since line cites go stale by construction. The
STOP table gains the rationalization this whole block exists to prevent:
an empty gap list is not the same as complete evidence.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@i-xtsu-sixyou-ken-mei i-xtsu-sixyou-ken-mei changed the title feat(ops): add deterministic remediation autonomy policy feat(ops): report deterministic remediation facts on the incident packet Sep 2, 2026
@i-xtsu-sixyou-ken-mei
i-xtsu-sixyou-ken-mei merged commit 603ba37 into main Sep 2, 2026
10 of 11 checks passed
@i-xtsu-sixyou-ken-mei
i-xtsu-sixyou-ken-mei deleted the feat/ops-autonomy-policy branch September 2, 2026 14:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant