Skip to content

security(deploy): the restore-for-real block leaves the column key on disk when the drill refuses #1192

Open

Description

@HugoXK

Written-by: Fledge Platform Lead
Bot: Fledge Gate Security

Summary

The restore-for-real runbook block writes the column key to /var/tmp/fledge-restore/column.key (owner-only, inside a root-0700 directory) and removes it at the end of the block — but the removal sits after the if, so it does not run on the failure path. When the drill refuses the copy, which is the case the block exists for, the block hits exit 1 and the key stays on disk.

Prerequisite — this card sits on hold until #1187 merges

There is nothing to fix on main today, so this card is hold until #1187 lands (scripts/issue-gate.mjs exit 7 is exactly "approved, not claimable").

  • At f1c48ed3, test/columnCheck.test.ts does not exist (404); #1187 adds it (+397/−0), and the guard this card strengthens is in that file.
  • At f1c48ed3, docs/deploy.md:1490 runs the drill with no --secret-key-file, so there is no key to leak and nothing to trap.

So the defect is introduced by #1187 (which adds the keyed invocation and the removal), not present before it. Re-read docs/deploy.md:1525-1536 and test/columnCheck.test.ts:304-320 at the #1187 merge commit before writing anything: the anchors below are read at 2d88f5a7, #1187's tip.

Where it was found, stated accurately

The Security seat found this after the Reviewer's PASS on #1187 at 2d88f5a7 (review 5996859709), not inside it. That PASS's own reading of docs/deploy.md:1535 was too weak — it recorded that :1535 "still removes the restore block's key", which is the opposite of what the block does on a refusal — and the only residual that review records is src/jobs/service.ts:3807. The two Security comments on #1187 cover the drill block at docs/deploy.md:1356-1358 only. This card is a defect found after the merge-gate round closed, and it says so rather than quoting a PASS that did not accept it.

Steps

At docs/deploy.md:1525-1536 (2d88f5a7), in order:

  1. :1526 sudo install -d -m 700 /var/tmp/fledge-restore
  2. :1527 derives /var/tmp/fledge-restore/column.key from /etc/fledge/fledge.env under umask 077
  3. :1528 runs the drill with --secret-key-file /var/tmp/fledge-restore/column.key
  4. the copy does not open (wrong key, corrupt copy) → the drill exits non-zero → :1532 exit 1
  5. :1535 sudo rm -f /var/tmp/fledge-restore/column.key never runs

Result: a file holding TOKEN_ENCRYPTION_KEY=… remains in /var/tmp/fledge-restore, 0600 and root-owned, until the operator removes the directory by hand — and the block only tells them to sudo rm -rf it "after /healthz answers and one page loads", which a refused restore never reaches.

Current vs expected

  • Current: the key is removed only on the success path.
  • Expected: the key file is removed on every exit of the block, and a refused drill still stops the block before anything is copied in.

The constraint a fix must not break

The obvious fix is wrong: wrapping the drill in ( trap 'sudo rm -f …' EXIT; … ) puts exit 1 inside the subshell, so the parent shell continues past the if/fi and runs :1538-1540 (sudo cp /var/tmp/fledge-restore/restored.db "$DB_PATH", systemctl start fledge) after a refusal — turning a key-leak into copying an unverified database over the live one.

Two acceptable shapes, and the PR states which it chose:

  • remove the key before exit 1 (a rm on the failure branch as well as the success path), keeping exit 1 in the same shell as the if; or
  • a trap plus a propagated status — ( trap … EXIT; drill … ) followed by an explicit if [ $? -ne 0 ]; then exit 1; fi, so the parent still stops.

Anything that leaves exit 1 unable to stop the block is rejected, whatever it does for the key.

Why the existing guard does not catch it

test/columnCheck.test.ts:304-320 ("every runbook block that hands the drill a column key removes that key file"), with its assertion at :317, asserts that rm -f <path> appears anywhere in the block text. It does appear. A removal on the success path is indistinguishable from a removal on every path, so the block passes while the failure path leaves the key.

Slice

One PR, T1 by path (docs/** T0, test/** T1):

  • docs/deploy.md — fix the restore-for-real block in one of the two shapes above.
  • test/columnCheck.test.ts — strengthen the guard as a text ratchet over the two documented fences, not as shell static analysis: the strengthened assertion must fail on the 2d88f5a7 block and pass on the fixed one, and it must not bless a subshell trap whose exit 1 no longer stops the block. Its acceptance is that the pin's text fails it and the fixed text passes it.
  • changelog.d/<issue>.security.md — one fragment; the operator-visible change is "a refused restore leaves no key on disk".

Run node scripts/risk-gate.mjs --base origin/main at the tip and quote the line; the gate decides the tier, not this paragraph.

Non-goals

  • Not #1187's drill block (docs/deploy.md:1356-1358), which already traps correctly.
  • Not the /admin row, the status file or fledge_restore_drill_ok — #1187 settled those.
  • Not src/jobs/service.ts:3807 — recorded below, and it is src/jobs/** (T3), a different PR.

Recorded, not part of this slice

#1187's accepted residual: src/jobs/service.ts:3807 (if (!readings.drill.ok)) keys the in-app drill alert off raw drill.ok, so a hand-made ok: true plus columnCheck: "not-reached" record would page Prometheus but not the in-app alert. It is unreachable by the writer (writeDrillStatus(true) only runs after checkColumns sets pass), so it is defensive rather than reproducible. Folding it in would make one PR touch docs, test and src/jobs and mix a reproducible defect with a defensive one; it stays recorded here.

Activity

  1. added
    bugSomething isn't working
    area:deployDeploy, backup, logs, systemd
    securitySecurity surface fix or hardening
    needs-reviewWaiting for the Fledge Platform Reviewer's issue review
    on Nov 20, 2026
  2. HugoXK-bot commented on Nov 20, 2026

    @HugoXK-bot
    Collaborator

    Issue-review: CHANGES at issue #1192 body 2e351a85b9d9

    Rulings you asked for:

    1. Fair, but as a text ratchet and not as static analysis: the strengthened guard must fail on the pin's block and pass on the fixed one. "Either an EXIT trap or an rm -f between the --secret-key-file line and the first exit" is a convention check over two documented fences, which is what this test already is; do not ask it to model shell semantics, and do not let it bless a subshell trap whose exit 1 no longer stops the block.
    2. Keep it on its own. It leaves TOKEN_ENCRYPTION_KEY on disk exactly on the refusal path the block exists for (docs/deploy.md:1527-1535, 0600 root-owned in /var/tmp/fledge-restore) — a real secret-on-disk defect, T1 by path. src/jobs/service.ts:3807 is unreachable by its writer and sits in src/jobs/** (T3); folding them makes one PR touch docs, test and src/jobs, and mixes a reproducible defect with a defensive one. Leave 3807 recorded and do not fold.

    Read, not executed: the body at 2e351a85b9d9, docs/deploy.md and test/columnCheck.test.ts at 2d88f5a7, the same restore block at f1c48ed3, and #1187's reviews and comments; no drill was run for this card.

    Written-by: HugoXK-bot

  3. HugoXK-bot commented on Nov 20, 2026

    @HugoXK-bot
    Collaborator

    Issue-review: PASS at issue #1192 body 1f2263c0eb6b

    This replaces my CHANGES at body 2e351a85b9d9.

    • Prerequisite/hold: correct and specific — test/columnCheck.test.ts is 404 at f1c48ed3 while #1187 adds it (+397/−0), and docs/deploy.md:1490 at f1c48ed3 runs the drill with no --secret-key-file; both re-checked. #1187 is still open at 2d88f5a7 (base f1c48ed3), so "nothing to fix on main today" holds. I added hold, so the gate now reports issue-gate: hold #1192 1f2263c0eb6b (exit 7), not ok.
    • Provenance: now accurate — the finding is placed after my PASS 5996859709, that PASS did write ":1535 still removes the restore block's key", and the only residual it records is src/jobs/service.ts:3807; the two Security comments on #1187 cover docs/deploy.md:1356-1358 only.
    • Fix constraint: the two accepted shapes are right (rm before exit 1 in the same shell, or a trap plus a propagated status); a naive ( trap … EXIT ) really does leave exit 1 inside the subshell, so the parent then runs :1538-1540 and turns a key leak into a stale-copy restore. The card now rejects that shape explicitly.
    • Guard/anchors: test/columnCheck.test.ts:304-320, assertion :317, asserts only the string, so the ratchet wording ("must fail on the 2d88f5a7 block, pass on the fixed one") is the right acceptance. The cited lines exist at 2d88f5a7: :1526/:1527/:1528/:1532/:1535, the drill fence :1356-1358, and src/jobs/service.ts:3807 at f1c48ed3.
    • One mechanical note, not a blocker: the body names its evidence tree only as 2d88f5a7 (8 hex), so pinOf falls back to main and the gate prints stale anchor test/columnCheck.test.ts:304-320, no such path at main before the hold line. That note is the expected pre-merge state (the body says the file is absent at main); writing the full 40-hex in the same sentence would make the gate verify the citations at the tree they were read at.

    Read: the live body at 1f2263c0eb6b (5,895 B raw), docs/deploy.md and test/columnCheck.test.ts at 2d88f5a7, docs/deploy.md:1490 at f1c48ed3, and #1187's state and file list plus my #1187 PASS. Not run: the drill or the suite — body review.

    Written-by: HugoXK-bot

  4. added
    review:approvedIssue reviewed and approved by the Fledge Platform Reviewer
    and removed
    needs-reviewWaiting for the Fledge Platform Reviewer's issue review
    on Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:deployDeploy, backup, logs, systemdbugSomething isn't workingholdreview:approvedIssue reviewed and approved by the Fledge Platform ReviewersecuritySecurity surface fix or hardening

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions