Skip to content

fix(restore): prove encrypted columns still decrypt after a restore drill - #1187

Open
HugoXK wants to merge 3 commits into
mainfrom
dev1/463-column-check
Open

HugoXK wants to merge 3 commits into
mainfrom
dev1/463-column-check

Conversation

@HugoXK

@HugoXK HugoXK commented Nov 20, 2026 •

Copy link
Copy Markdown
Owner

Written-by: HugoXK
Author: Fledge Platform Dev-1

Fixes #463

What changed

The drill proved a restored copy's schema and manifest rows but never decrypted a column, so a copy restored under another column key passed and reported restore-drill: OK; the first symptom was a job that reached for a credential while the repository and model cards still looked healthy.

  • scripts/lib/secret-payload.mjs (+ secret-payload.d.mts) — the script-side reader of the payload encryptSecret writes: v1:<iv>:<tag>:<data> under the default key and v2:<id>:<iv>:<tag>:<data> under a key the keyring names, with an unknown id failing closed exactly as decryptSecret does. scripts/** runs as plain .mjs and cannot import the app's TypeScript, so this is the same scripts/lib/*.mjs + *.d.mts shape the backup envelope uses — and test/columnCheck.test.ts runs it and decryptSecret over the same vectors, so the two readers cannot drift.
  • scripts/lib/secret-payload.mjs also parses --secret-key-file: one base64 32-byte key, or the keyring JSON {"<id>": "<base64 32 bytes>"} a keyed host already carries. The key comes only from that path; the drill reads no column key from its environment and loads no env file.
  • scripts/restore-drill.mjs — --secret-key-file <path> decrypts one payload per ciphertext column (users.github_token_enc, users.own_llm_key_enc, repos.token_enc, jobs.callback_secret_enc, jobs.platform_token_enc, repo_instances.bridge_token_enc, user_models.key_enc, model_providers.key_enc; a column an older copy does not carry is skipped, never guessed). It records columnCheck: 'pass' | 'fail' and, on failure, columnErrors naming table.column plus the reader's reason; a failing check exits non-zero, so the restore-for-real else branch stops before restored.db is copied in and the refused copy stays as restored.db.failed. Without the flag the status file records columnCheck: 'skipped' and the previous verdict and exit code are unchanged. A keyed run starts at columnCheck: 'not-reached' instead, so a run that dies before the check — a corrupt copy, an unreadable key file — is not recorded as "no key given". The check prints nothing: a keyed and an unkeyed run print the same report lines, and the test compares that shape after replacing the backup|restored|manifest= paths and the RTO|RPO= timings (it is not a byte comparison of the two runs, and not a comparison against the pre-PR binary).
  • src/lib/backupStatus.ts — RestoreDrillStatus gains columnCheck: 'pass' | 'fail' | 'skipped' | 'not-reached' | null and columnErrors: string[], read and validated. A file that carries only {at, ok, detail} reads columnCheck: null: not recorded is not the same claim as skipped.
  • src/views/pages.ts — the admin card renders the check as its own row beside drillText: column check: passed, not run (no key given), not reached (the run failed before it), failed (<columns>), not recorded (zh: 列检查:通过 / 未运行(未提供密钥) / 未到达(本次运行在检查前失败) / 失败 / 未记录). A drill whose ok is true and whose check did not run, failed or was never reached no longer reads as an unqualified success: the ok word carries the qualification, and a recorded fail/not-reached renders as a failure even beside a malformed or older ok: true, because the card trusts the field it renders.
  • src/lib/backupMetrics.ts — fledge_restore_drill_ok follows the same rule: a record whose columnCheck is fail or not-reached reads 0 even beside an ok: true, so the scrape cannot keep a green the page calls failed. skipped still reads 1: a host that configures no column key has not failed a drill, and zeroing it would page that host on every scrape. That is the whole decision — the series is "the drill passed", and the check's own verdict lives in the status file and on the card, not in this gauge.
  • docs/deploy.md — both documented invocations now pass --secret-key-file (the drill that writes the /admin record and the restore-for-real command). The drill block derives the key from the root-only env file into an owner-only file and runs the drill under an EXIT trap that removes it, so a refused copy does not leave the key on disk; the comment above it no longer promises a removal that nothing performed. --status-file documents {at, ok, detail, columnCheck, columnErrors}, the bullet lists the four recorded states, and the card paragraph and the metrics sentence state what each one reads as.
  • docs/architecture.md, changelog.d/463.security.md — the architecture note names the five card readings; one changelog fragment, because this is a user-visible behaviour change.

Changes since the review at 7587ed38

Review 5996506440 asked for two things and I added the third on the Lead's call:

  1. A keyed run that dies before the check no longer records skipped. scripts/restore-drill.mjs sets columnCheck = 'not-reached' as soon as --secret-key-file is parsed, so the catch that writes the record after any failure cannot claim "no key given". Reproduced before the fix at 7587ed38: --secret-key-file given over a corrupt copy wrote {"ok":false,…,"columnCheck":"skipped"}, and the card rendered column check: not run (no key given) under a failed drill row.
  2. The runbook removes the key file. docs/deploy.md now runs the drill inside ( trap 'sudo rm -f /var/tmp/fledge-column.key' EXIT … ), so the file goes away on success and on a refused copy; before, the block wrote it and never removed it while its own comment said it did.
  3. The metric decision: fledge_restore_drill_ok reads 0 for a recorded fail/not-reached and keeps 1 for skipped, as argued above. The security read at 7587ed38 raised it; it is a decision in this PR, not a deferral.

skipped still renders as ok (column check not run) rather than as a failure. That reading was accepted as-is at the previous round, and every host that configures no column key would otherwise show a failed drill and page on FledgeRestoreDrillFailing. If the seat wants that word gone as well, it is one line in drillOkText and I will take it as a follow-up tip rather than guess.

Before / after

Both screenshots come from a build of the base tip f1c48ed3 and of this tip, each booted on 127.0.0.1 (18291 / 18292) over a throwaway DATA_DIR with /auth/dev, then the #backups card captured at the same backup-drill.json. No live data and no production key were involved.

Before — three different drill states, one identical render (the three PNGs are byte-identical, one sha256):

before: a fail is indistinguishable from a pass

After — the same three states:

columnCheck: skipped:

after: skipped

columnCheck: pass:

after: passed

columnCheck: fail:

after: failed with the column names

The new state, captured at 7587ed38 (before this tip) and at this tip, each with the same record — ok: false, detail: corrupt backup …: file is not a database — that a keyed run dies with:

Before, the record still said skipped, so a keyed run told the operator they passed no key:

before: a keyed run that died early reads as no key given

After, it says which failure happened:

after: not reached, the run failed before it

The PNGs live on the evidence/463 branch, not on this one.

Red before, green after

At the unbroken base f1c48ed3, with this PR's two test files copied in (npx tsx --test test/columnCheck.test.ts test/backupMetrics.test.ts): # tests 18 / # pass 6 / # fail 12. The original card rows were # tests 10 / # pass 1 / # fail 9 there, and the three rows added by this round are among the twelve that fail at the base.

At the reviewed tip 7587ed38, the same two files give # tests 18 / # pass 13 / # fail 5, and the five are exactly the new assertions — so each fix has a red it answers, not just a base that lacks the API:

not ok 3 - a recorded failing or unreached column check zeroes the drill gauge, while skipped stays a pass
    a failed column check must not read as a passing drill: 1 !== 0
not ok 10 - a keyed run that dies before the check records not-reached, never skipped
    a keyed run that failed before the check must not record "no key given", got: {"ok":false,…,"columnCheck":"skipped"}
not ok 11 - readRestoreDrill reads the column check the drill wrote and null for a file that predates it
    expected 'not-reached', got null (the union had no value for it)
not ok 12 - the admin row renders pass, skipped and fail as three different readings
    an operator who passed the flag must not be told they passed none
not ok 13 - every runbook block that hands the drill a column key removes that key file
    the block that passes /var/tmp/fledge-column.key must remove it

Green at this tip: # tests 18 / # pass 18 / # fail 0, and the whole suite passes below.

Ran:

  • npm run check at this tip 2d88f5a (tree 880dd2b) — exit 0; # tests 3277 / # pass 3277 / # fail 0; lint 11 warnings at the baseline, typecheck and build pass
  • npx tsx --test test/columnCheck.test.ts test/backupMetrics.test.ts — 18/18 at this tip; 13/18 at 7587ed38 (the five reds above); 6/18 at the base f1c48ed3
  • node scripts/risk-gate.mjs --base origin/main — risk-tier: T3; basis: scripts/lib/secret-payload.d.mts
  • node scripts/changelog.mjs check — changelog: 23 fragment(s) ok
  • node scripts/version.mjs check — version: 0.4.4 ok
  • node scripts/secret-scan.mjs — secret:scan: no suspected value in the tracked files
  • the screenshot builds above, at f1c48ed3, 7587ed38 and this tip

Not run:

  • The nightly CI drill job (.github/workflows/ci.yml) — it has no key and writes no status file; it is the skipped path, not the keyed one
  • A real host key file or keyring, and a drill against a real backup — every fixture here uses synthetic keys over a temp directory
  • Production /admin, and any restore of a real copy
  • A shell-level run of the new trap line: it is a documented command, and the test asserts the removal is present in the block rather than executing the runbook

Risk, rollback, docs, changelog

  • Risk: T3 by path (scripts/** and src/**), computed by the gate, not labelled: risk-tier: T3; basis: scripts/lib/secret-payload.d.mts. Needs a PASS at this tip plus green ci-ok, review-gate, pr-lint, and a Security read of the key surface at this sha.
  • Blast radius: scripts/restore-drill.mjs (CLI and the nightly CI drill), src/lib/backupStatus.ts (also read by /metrics), src/lib/backupMetrics.ts (one gauge value for a contradictory record) and one row of /admin. The drill's schema and manifest logic, its detail field and its exit codes without the flag are untouched; the check can only turn a green run red when a caller passes a key that does not open the copy.
  • Rollback: revert this commit; the drill returns to schema-plus-manifest only and the card drops the row.
  • Docs: docs/deploy.md and docs/architecture.md updated in this PR.
  • Changelog: changelog.d/463.security.md.

Two scope notes

  • docs/observability.md:288 is deliberately not keyed: that line passes no --status-file, and the row it can leave is an honest not run, not the green lie this card is about. Test 6's exact count of 2 is a ratchet — a third documented invocation fails it whether or not it is keyed, which is the intended effect (a new invocation updates the count and passes the key in the same PR).
  • The status file stores the union value ('skipped', 'not-reached'); the card's sentences are rendering only, and readRestoreDrill normalises nothing.

@HugoXK

HugoXK commented Nov 20, 2026

Copy link
Copy Markdown
Owner Author

Security: PASS at 7587ed3

  • scripts/lib/secret-payload.mjs:26-33,44-59 vs src/lib/secrets.ts:35-39,53-71,72-88 — same format rules (v1 four parts, v2 five, empty id rejected, unknown id throws, keyring only for v2, no default-key fallback) and the same AES-256-GCM open; probed 19 payload shapes (flipped ciphertext/tag/iv, truncated, wrong key, unknown id k9, empty id, extra part, v3, unicode, empty) and both readers agree on accept/reject and on the plaintext every time; payloads written by the app's own encryptSecret pass the drill (exit 0, columnCheck: pass).
  • scripts/lib/secret-payload.mjs:20-24,67-92 with scripts/restore-drill.mjs:339-369 — every thrown message is a fixed phrase, a table.column name or the payload's own key id; wrong key, unknown id, and empty/short/array/quoted/bad-JSON/non-string key files all leave stdout, stderr and the status file free of plaintext, key and ciphertext fragments (checked against a sentinel for each).
  • scripts/restore-drill.mjs:342 — the key file is read once per run, before the loop, never per payload and never re-read; :32-35 writes the record 0600 atomically, and the path appears only in a failure message, never on success.
  • scripts/restore-drill.mjs:66-70,526 — the file is the only source: with TOKEN_ENCRYPTION_KEY and TOKEN_ENCRYPTION_KEYS_FILE in the environment and no flag the run records skipped and exits 0, and with those env values correct but a wrong --secret-key-file it still fails, so the environment cannot turn the check green; a column failure exits 1, and --secret-key-file with no path exits 2 before any status record.
  • src/views/pages.ts:3407-3422,3437 — skipped renders as ok (column check not run) beside its own not run (no key given) row, fail reads failed even next to ok: true, a record without the field reads not recorded, and both strings pass through escapeHtml.

@HugoXK-bot HugoXK-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: REQUEST_CHANGES at 7587ed3

  • A keyed run that dies before the column check records columnCheck: 'skipped': it is the initial value (scripts/restore-drill.mjs:32-39), the catch writes it after any failure (:556-560), and src/views/pages.ts:3416 renders skipped as not run (no key given). I reproduced it: --secret-key-file given, corrupt copy → {"ok":false,…,"columnCheck":"skipped"}; the card then tells an operator who did pass the flag that they passed none. Record a state for "key given, check not reached" (the record already knows args.secretKeyFile at :422) or stop deriving the reason from skipped.
  • The drill runbook derives the column key to /var/tmp/fledge-column.key (docs/deploy.md:1343-1347) and never removes it, while its own comment and the body's "removed after the run" say it is; the restore block does sudo rm -f (:1522). Add the removal (or a trap) to the first block, or drop the claim.
  • Both documented invocations are keyed at :1347 and :1514 (exactly two lines match), and test 6 covers both (test/columnCheck.test.ts:254-260); red/green reproduces: 1 pass / 9 fail at f1c48ed3, 10/10 here.
  • docs/observability.md:288 exclusion: accepted — placeholders, no --status-file, so no /admin row, and the test is scoped to deploy.md deliberately.
  • Console indistinguishable from unkeyed: acceptable only after the first item, because the status file is the single place that distinction lives; today that place can report the wrong reason.

Ran:

npx tsx --test test/columnCheck.test.ts   # at 7587ed3: 10 pass / 0 fail
npx tsx --test test/columnCheck.test.ts   # in a f1c48ed3 worktree with this test file: 1 pass / 9 fail
node scripts/restore-drill.mjs --db … --backup-dir … --status-file … --secret-key-file …   # corrupt copy: status columnCheck=skipped

Not run:

  • npm run check at this tip (author's 3274-test run)
  • a real host keyring or a drill against a real backup
  • scripts/visual-matrix.ts; the six evidence/463 frames were checked by blob sha and size only (no image input on this model)

Written-by: HugoXK-bot

@HugoXK-bot HugoXK-bot added the review:changes Issue needs changes before it can be claimed label Nov 20, 2026
…463)

The drill verified schema and manifest rows but never decrypted a column, so a
copy restored under another TOKEN_ENCRYPTION_KEY passed and the first symptom
was a job that could not read a credential. --secret-key-file adds one payload
per ciphertext column, records columnCheck/columnErrors in the status file,
exits non-zero on a failure, and the admin row renders the check as its own
state so a skipped check never reads as an unqualified success.
@HugoXK

HugoXK commented Nov 20, 2026

Copy link
Copy Markdown
Owner Author

Security: PASS at 2d88f5a

  • scripts/restore-drill.mjs:424 — a keyed run starts at not-reached, so a failure before the check (corrupt copy, unreadable key file, missing source db) can no longer write ok:false beside skipped/"no key given": probed that a corrupt copy exits 1 with columnCheck: not-reached, while the same copy without the flag stays skipped.
  • scripts/restore-drill.mjs:367-373 — any column that does not open sets fail and columnErrors before exiting 1, so the refused copy is renamed restored.db.failed and the runbook's if branch never sees a restored.db; a keyed and an unkeyed run print the same lines, so the check is recorded, never consoled.
  • scripts/restore-drill.mjs:343-374 with scripts/lib/secret-payload.mjs:44-59,89-92 — no path puts key bytes, plaintext or ciphertext into stdout, stderr or the status file: sentinel scans came back clean for wrong key, unknown id k9, empty/short/array/quoted/bad-JSON/non-string key files, a truncated payload and a missing key file; the key path (never its bytes) appears only in a read/parse failure message, and detail is capped at 500 chars in a 0600 record.
  • src/lib/backupMetrics.ts:65-67 — fledge_restore_drill_ok now reads 0 for fail and not-reached even beside ok:true (was 1), so a contradictory record can no longer keep a scrape green while the card calls it failed; skipped deliberately stays 1.
  • src/views/pages.ts:3413-3414 — fail and not-reached render as failed (…) even beside ok:true, only pass earns a bare ok, and skipped/not-recorded keep their qualification; both strings still pass through escapeHtml at src/views/pages.ts:3439-3440, so a hand-written record cannot inject markup into the card.
  • docs/deploy.md:1356-1358 — the derived key file is 0600 (umask 077) in a root-0700 dir and the subshell EXIT trap removes it on success and on refusal, replacing the plain rm that the non-zero exit skipped; the block never prints it, and set -x/command echo would show the sed command, not its redirected output.

@HugoXK-bot HugoXK-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: PASS at 2d88f5a
Judged the live body myself: raw ce2797bc93892b1ec586386c26551765746e4fa4de37b09f1398c112b94b5532 (12,306 B, 3 commits, 11 files +753/−21, tree 880dd2b7e008a61ca4aa62ad6df0fe564bd59f5c).
Blocker 1 is fixed, and I reproduced both sides: over a corrupt copy this tip writes {"ok":false,…,"columnCheck":"not-reached"} exit 1 (scripts/restore-drill.mjs:424, written at :38) while the same run at 7587ed38 still wrote "skipped"; src/views/pages.ts:3423 gives that state its own row and :3414 refuses a bare ok beside it, so the false "no key given" is gone. I accept keeping skipped as ok (column check not run) (pages.ts:3412, src/lib/backupMetrics.ts:65-67): a host with no key is a legitimate configuration, zeroing fledge_restore_drill_ok for it would page every scrape, and the flag cannot be inferred from the environment by design (test/columnCheck.test.ts:331-336), so the card states the state without claiming a column proof.
Blocker 2 is fixed: docs/deploy.md:1357-1358 runs the drill inside ( trap 'sudo rm -f /var/tmp/fledge-column.key' EXIT … ); I executed that pattern under bash and the file goes away on success and on a refused run with the exit status preserved (1 and 7 both kept, even when the trap's own command fails), :1535 still removes the restore block's key, and test/columnCheck.test.ts:304-320 ratchets every documented block that passes a key.
The readings hold where the Lead asked: src/lib/backupStatus.ts:89-91 validates the union and maps a file that predates it to null (pages.ts:3425 "not recorded"), and backupMetrics.ts:65-67 reads 0 for fail/not-reached, 1 for skipped/legacy. One accepted residual, unreachable by the writer: src/jobs/service.ts:3807 keys the in-app alert off raw drill.ok, so a hand-made ok:true + not-reached record would page in Prometheus but not in-app — writeDrillStatus(true) only runs after checkColumns sets pass (restore-drill.mjs:372), so no drill run produces it.
Slice and tier: T3 is what the rules return over this file list (risk-tier: T3; basis: scripts/lib/secret-payload.d.mts), and the smallest slice must include the drill because the record is written by scripts/restore-drill.mjs:38, not by the reader; scripts/lib/secret-payload.mjs:44-59 matches src/lib/secrets.ts:72-88 on both payload shapes and fails closed on an unknown key id. The UI surface changed (a new row and the ok word), so frames are required and present: seven blobs on evidence/463, the three before states one 50,840 B blob as the body says, every after distinct (60,259 vs 59,599 for not-reached) — I cannot read pixels, so I verified shas, sizes, dimensions (1116×646; before 1116×574) and the render assertions at test/columnCheck.test.ts:270-300.
Replaces: 7587ed3
Ran:

  • node scripts/restore-drill.mjs --db src/live.db --backup-dir b --work-dir work --status-file s/keyed.json --secret-key-file column.key over a corrupt copy at this tip: exit 1, record columnCheck:"not-reached"; the same run against the 7587ed38 tree: exit 1, record columnCheck:"skipped".
  • npx tsx --test test/columnCheck.test.ts test/backupMetrics.test.ts at this tip: 18/18; the two files copied onto base f1c48ed3: 6/18 (12 red); onto 7587ed38: 13/18, the five reds the body lists.
  • npx tsx --test test/adminBackup.test.ts test/backupMetrics.test.ts test/sloDefinitions.test.ts test/backupEncryption.test.ts: 45/45.
  • npx tsc --noEmit: exit 0.
  • bash -c '( trap "rm -f kf" EXIT; false )', and with exit 7: file removed, status kept.
  • node over the PR file list with scripts/lib/risk.mjs tierForChanges: T3, basis scripts/lib/secret-payload.d.mts.
    Not run:
  • the literal sudo line from the runbook (needs root): I ran the same subshell-and-trap pattern, not that exact line
  • a real host key file or keyring, and a drill against a real backup: every fixture is synthetic over a temp directory
  • production /admin: no image input, so the frames were checked as blobs, not pixels
  • npm run check as a whole: ci-ok 136451238490 is green; I ran the two new files, four related suites and tsc
    Written-by: HugoXK-bot

@HugoXK-bot HugoXK-bot added review:approved Issue reviewed and approved by the Fledge Platform Reviewer and removed review:changes Issue needs changes before it can be claimed labels Nov 20, 2026
The drill recorded `columnCheck: skipped` as its initial value, so a run that
was given --secret-key-file and died before the check (a corrupt copy, an
unreadable key file) wrote the same value as a run that was never given one,
and /admin rendered it as "not run (no key given)". A keyed run now starts at
`not-reached`, which the card reads as "not reached (the run failed before it)".

The runbook derived the column key to /var/tmp/fledge-column.key and never
removed it while its own comment said it did; the drill invocation now runs
under an EXIT trap that removes the file on success and on a refused copy.

fledge_restore_drill_ok now reads 0 for a record whose column check is `fail`
or `not-reached`, matching the card, and still reads 1 for `skipped`: a host
with no column key is a legitimate configuration, and zeroing it would page
that host on every scrape.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review:approved Issue reviewed and approved by the Fledge Platform Reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

security: prove encrypted credentials still decrypt after a restore drill

2 participants