Repository navigation
Conversation
|
Security: PASS at 7587ed3
|
HugoXK-bot
left a comment
There was a problem hiding this comment.
Verdict: REQUEST_CHANGES at 7587ed3
- A keyed run that dies before the column check records
columnCheck: 'skipped': it is the initial value (scripts/restore-drill.mjs:32-39), the catch writes it after any failure (:556-560), andsrc/views/pages.ts:3416rendersskippedasnot run (no key given). I reproduced it:--secret-key-filegiven, corrupt copy →{"ok":false,…,"columnCheck":"skipped"}; the card then tells an operator who did pass the flag that they passed none. Record a state for "key given, check not reached" (the record already knowsargs.secretKeyFileat:422) or stop deriving the reason fromskipped. - The drill runbook derives the column key to
/var/tmp/fledge-column.key(docs/deploy.md:1343-1347) and never removes it, while its own comment and the body's "removed after the run" say it is; the restore block doessudo rm -f(:1522). Add the removal (or a trap) to the first block, or drop the claim. - Both documented invocations are keyed at
:1347and:1514(exactly two lines match), and test 6 covers both (test/columnCheck.test.ts:254-260); red/green reproduces: 1 pass / 9 fail atf1c48ed3, 10/10 here. docs/observability.md:288exclusion: accepted — placeholders, no--status-file, so no/adminrow, and the test is scoped todeploy.mddeliberately.- Console indistinguishable from unkeyed: acceptable only after the first item, because the status file is the single place that distinction lives; today that place can report the wrong reason.
Ran:
npx tsx --test test/columnCheck.test.ts # at 7587ed3: 10 pass / 0 fail
npx tsx --test test/columnCheck.test.ts # in a f1c48ed3 worktree with this test file: 1 pass / 9 fail
node scripts/restore-drill.mjs --db … --backup-dir … --status-file … --secret-key-file … # corrupt copy: status columnCheck=skipped
Not run:
npm run checkat this tip (author's 3274-test run)- a real host keyring or a drill against a real backup
scripts/visual-matrix.ts; the sixevidence/463frames were checked by blob sha and size only (no image input on this model)
Written-by: HugoXK-bot
…463) The drill verified schema and manifest rows but never decrypted a column, so a copy restored under another TOKEN_ENCRYPTION_KEY passed and the first symptom was a job that could not read a credential. --secret-key-file adds one payload per ciphertext column, records columnCheck/columnErrors in the status file, exits non-zero on a failure, and the admin row renders the check as its own state so a skipped check never reads as an unqualified success.
|
Security: PASS at 2d88f5a
|
HugoXK-bot
left a comment
There was a problem hiding this comment.
Verdict: PASS at 2d88f5a
Judged the live body myself: raw ce2797bc93892b1ec586386c26551765746e4fa4de37b09f1398c112b94b5532 (12,306 B, 3 commits, 11 files +753/−21, tree 880dd2b7e008a61ca4aa62ad6df0fe564bd59f5c).
Blocker 1 is fixed, and I reproduced both sides: over a corrupt copy this tip writes {"ok":false,…,"columnCheck":"not-reached"} exit 1 (scripts/restore-drill.mjs:424, written at :38) while the same run at 7587ed38 still wrote "skipped"; src/views/pages.ts:3423 gives that state its own row and :3414 refuses a bare ok beside it, so the false "no key given" is gone. I accept keeping skipped as ok (column check not run) (pages.ts:3412, src/lib/backupMetrics.ts:65-67): a host with no key is a legitimate configuration, zeroing fledge_restore_drill_ok for it would page every scrape, and the flag cannot be inferred from the environment by design (test/columnCheck.test.ts:331-336), so the card states the state without claiming a column proof.
Blocker 2 is fixed: docs/deploy.md:1357-1358 runs the drill inside ( trap 'sudo rm -f /var/tmp/fledge-column.key' EXIT … ); I executed that pattern under bash and the file goes away on success and on a refused run with the exit status preserved (1 and 7 both kept, even when the trap's own command fails), :1535 still removes the restore block's key, and test/columnCheck.test.ts:304-320 ratchets every documented block that passes a key.
The readings hold where the Lead asked: src/lib/backupStatus.ts:89-91 validates the union and maps a file that predates it to null (pages.ts:3425 "not recorded"), and backupMetrics.ts:65-67 reads 0 for fail/not-reached, 1 for skipped/legacy. One accepted residual, unreachable by the writer: src/jobs/service.ts:3807 keys the in-app alert off raw drill.ok, so a hand-made ok:true + not-reached record would page in Prometheus but not in-app — writeDrillStatus(true) only runs after checkColumns sets pass (restore-drill.mjs:372), so no drill run produces it.
Slice and tier: T3 is what the rules return over this file list (risk-tier: T3; basis: scripts/lib/secret-payload.d.mts), and the smallest slice must include the drill because the record is written by scripts/restore-drill.mjs:38, not by the reader; scripts/lib/secret-payload.mjs:44-59 matches src/lib/secrets.ts:72-88 on both payload shapes and fails closed on an unknown key id. The UI surface changed (a new row and the ok word), so frames are required and present: seven blobs on evidence/463, the three before states one 50,840 B blob as the body says, every after distinct (60,259 vs 59,599 for not-reached) — I cannot read pixels, so I verified shas, sizes, dimensions (1116×646; before 1116×574) and the render assertions at test/columnCheck.test.ts:270-300.
Replaces: 7587ed3
Ran:
node scripts/restore-drill.mjs --db src/live.db --backup-dir b --work-dir work --status-file s/keyed.json --secret-key-file column.keyover a corrupt copy at this tip: exit 1, recordcolumnCheck:"not-reached"; the same run against the7587ed38tree: exit 1, recordcolumnCheck:"skipped".npx tsx --test test/columnCheck.test.ts test/backupMetrics.test.tsat this tip: 18/18; the two files copied onto basef1c48ed3: 6/18 (12 red); onto7587ed38: 13/18, the five reds the body lists.npx tsx --test test/adminBackup.test.ts test/backupMetrics.test.ts test/sloDefinitions.test.ts test/backupEncryption.test.ts: 45/45.npx tsc --noEmit: exit 0.bash -c '( trap "rm -f kf" EXIT; false )', and withexit 7: file removed, status kept.nodeover the PR file list withscripts/lib/risk.mjstierForChanges: T3, basisscripts/lib/secret-payload.d.mts.
Not run:- the literal
sudoline from the runbook (needs root): I ran the same subshell-and-trap pattern, not that exact line - a real host key file or keyring, and a drill against a real backup: every fixture is synthetic over a temp directory
- production
/admin: no image input, so the frames were checked as blobs, not pixels npm run checkas a whole:ci-ok136451238490is green; I ran the two new files, four related suites andtsc
Written-by: HugoXK-bot
The drill recorded `columnCheck: skipped` as its initial value, so a run that was given --secret-key-file and died before the check (a corrupt copy, an unreadable key file) wrote the same value as a run that was never given one, and /admin rendered it as "not run (no key given)". A keyed run now starts at `not-reached`, which the card reads as "not reached (the run failed before it)". The runbook derived the column key to /var/tmp/fledge-column.key and never removed it while its own comment said it did; the drill invocation now runs under an EXIT trap that removes the file on success and on a refused copy. fledge_restore_drill_ok now reads 0 for a record whose column check is `fail` or `not-reached`, matching the card, and still reads 1 for `skipped`: a host with no column key is a legitimate configuration, and zeroing it would page that host on every scrape.
Written-by: HugoXK
Author: Fledge Platform Dev-1
Fixes #463
What changed
The drill proved a restored copy's schema and manifest rows but never decrypted a column, so a copy restored under another column key passed and reported
restore-drill: OK; the first symptom was a job that reached for a credential while the repository and model cards still looked healthy.scripts/lib/secret-payload.mjs(+secret-payload.d.mts) — the script-side reader of the payloadencryptSecretwrites:v1:<iv>:<tag>:<data>under the default key andv2:<id>:<iv>:<tag>:<data>under a key the keyring names, with an unknown id failing closed exactly asdecryptSecretdoes.scripts/**runs as plain.mjsand cannot import the app's TypeScript, so this is the samescripts/lib/*.mjs+*.d.mtsshape the backup envelope uses — andtest/columnCheck.test.tsruns it anddecryptSecretover the same vectors, so the two readers cannot drift.scripts/lib/secret-payload.mjsalso parses--secret-key-file: one base64 32-byte key, or the keyring JSON{"<id>": "<base64 32 bytes>"}a keyed host already carries. The key comes only from that path; the drill reads no column key from its environment and loads no env file.scripts/restore-drill.mjs—--secret-key-file <path>decrypts one payload per ciphertext column (users.github_token_enc,users.own_llm_key_enc,repos.token_enc,jobs.callback_secret_enc,jobs.platform_token_enc,repo_instances.bridge_token_enc,user_models.key_enc,model_providers.key_enc; a column an older copy does not carry is skipped, never guessed). It recordscolumnCheck: 'pass' | 'fail'and, on failure,columnErrorsnamingtable.columnplus the reader's reason; a failing check exits non-zero, so the restore-for-realelsebranch stops beforerestored.dbis copied in and the refused copy stays asrestored.db.failed. Without the flag the status file recordscolumnCheck: 'skipped'and the previous verdict and exit code are unchanged. A keyed run starts atcolumnCheck: 'not-reached'instead, so a run that dies before the check — a corrupt copy, an unreadable key file — is not recorded as "no key given". The check prints nothing: a keyed and an unkeyed run print the same report lines, and the test compares that shape after replacing thebackup|restored|manifest=paths and theRTO|RPO=timings (it is not a byte comparison of the two runs, and not a comparison against the pre-PR binary).src/lib/backupStatus.ts—RestoreDrillStatusgainscolumnCheck: 'pass' | 'fail' | 'skipped' | 'not-reached' | nullandcolumnErrors: string[], read and validated. A file that carries only{at, ok, detail}readscolumnCheck: null: not recorded is not the same claim as skipped.src/views/pages.ts— the admin card renders the check as its own row besidedrillText:column check: passed,not run (no key given),not reached (the run failed before it),failed (<columns>),not recorded(zh:列检查:通过 / 未运行(未提供密钥) / 未到达(本次运行在检查前失败) / 失败 / 未记录). A drill whoseokis true and whose check did not run, failed or was never reached no longer reads as an unqualified success: theokword carries the qualification, and a recordedfail/not-reachedrenders as a failure even beside a malformed or olderok: true, because the card trusts the field it renders.src/lib/backupMetrics.ts—fledge_restore_drill_okfollows the same rule: a record whosecolumnCheckisfailornot-reachedreads0even beside anok: true, so the scrape cannot keep a green the page calls failed.skippedstill reads1: a host that configures no column key has not failed a drill, and zeroing it would page that host on every scrape. That is the whole decision — the series is "the drill passed", and the check's own verdict lives in the status file and on the card, not in this gauge.docs/deploy.md— both documented invocations now pass--secret-key-file(the drill that writes the/adminrecord and the restore-for-real command). The drill block derives the key from the root-only env file into an owner-only file and runs the drill under anEXITtrap that removes it, so a refused copy does not leave the key on disk; the comment above it no longer promises a removal that nothing performed.--status-filedocuments{at, ok, detail, columnCheck, columnErrors}, the bullet lists the four recorded states, and the card paragraph and the metrics sentence state what each one reads as.docs/architecture.md,changelog.d/463.security.md— the architecture note names the five card readings; one changelog fragment, because this is a user-visible behaviour change.Changes since the review at
7587ed38Review
5996506440asked for two things and I added the third on the Lead's call:skipped.scripts/restore-drill.mjssetscolumnCheck = 'not-reached'as soon as--secret-key-fileis parsed, so the catch that writes the record after any failure cannot claim "no key given". Reproduced before the fix at7587ed38:--secret-key-filegiven over a corrupt copy wrote{"ok":false,…,"columnCheck":"skipped"}, and the card renderedcolumn check: not run (no key given)under afaileddrill row.docs/deploy.mdnow runs the drill inside( trap 'sudo rm -f /var/tmp/fledge-column.key' EXIT … ), so the file goes away on success and on a refused copy; before, the block wrote it and never removed it while its own comment said it did.fledge_restore_drill_okreads0for a recordedfail/not-reachedand keeps1forskipped, as argued above. The security read at7587ed38raised it; it is a decision in this PR, not a deferral.skippedstill renders asok (column check not run)rather than as a failure. That reading was accepted as-is at the previous round, and every host that configures no column key would otherwise show a failed drill and page onFledgeRestoreDrillFailing. If the seat wants that word gone as well, it is one line indrillOkTextand I will take it as a follow-up tip rather than guess.Before / after
Both screenshots come from a build of the base tip
f1c48ed3and of this tip, each booted on127.0.0.1(18291 / 18292) over a throwawayDATA_DIRwith/auth/dev, then the#backupscard captured at the samebackup-drill.json. No live data and no production key were involved.Before — three different drill states, one identical render (the three PNGs are byte-identical, one sha256):
After — the same three states:
columnCheck: skipped:columnCheck: pass:columnCheck: fail:The new state, captured at
7587ed38(before this tip) and at this tip, each with the same record —ok: false,detail: corrupt backup …: file is not a database— that a keyed run dies with:Before, the record still said
skipped, so a keyed run told the operator they passed no key:After, it says which failure happened:
The PNGs live on the
evidence/463branch, not on this one.Red before, green after
At the unbroken base
f1c48ed3, with this PR's two test files copied in (npx tsx --test test/columnCheck.test.ts test/backupMetrics.test.ts):# tests 18 / # pass 6 / # fail 12. The original card rows were# tests 10 / # pass 1 / # fail 9there, and the three rows added by this round are among the twelve that fail at the base.At the reviewed tip
7587ed38, the same two files give# tests 18 / # pass 13 / # fail 5, and the five are exactly the new assertions — so each fix has a red it answers, not just a base that lacks the API:Green at this tip:
# tests 18 / # pass 18 / # fail 0, and the whole suite passes below.Ran:
npm run checkat this tip2d88f5a(tree880dd2b) — exit 0;# tests 3277 / # pass 3277 / # fail 0; lint 11 warnings at the baseline, typecheck and build passnpx tsx --test test/columnCheck.test.ts test/backupMetrics.test.ts— 18/18 at this tip; 13/18 at7587ed38(the five reds above); 6/18 at the basef1c48ed3node scripts/risk-gate.mjs --base origin/main—risk-tier: T3; basis: scripts/lib/secret-payload.d.mtsnode scripts/changelog.mjs check—changelog: 23 fragment(s) oknode scripts/version.mjs check—version: 0.4.4 oknode scripts/secret-scan.mjs—secret:scan: no suspected value in the tracked filesf1c48ed3,7587ed38and this tipNot run:
.github/workflows/ci.yml) — it has no key and writes no status file; it is theskippedpath, not the keyed one/admin, and any restore of a real copytrapline: it is a documented command, and the test asserts the removal is present in the block rather than executing the runbookRisk, rollback, docs, changelog
scripts/**andsrc/**), computed by the gate, not labelled:risk-tier: T3; basis: scripts/lib/secret-payload.d.mts. Needs a PASS at this tip plus greenci-ok,review-gate,pr-lint, and a Security read of the key surface at this sha.scripts/restore-drill.mjs(CLI and the nightly CI drill),src/lib/backupStatus.ts(also read by/metrics),src/lib/backupMetrics.ts(one gauge value for a contradictory record) and one row of/admin. The drill's schema and manifest logic, itsdetailfield and its exit codes without the flag are untouched; the check can only turn a green run red when a caller passes a key that does not open the copy.docs/deploy.mdanddocs/architecture.mdupdated in this PR.changelog.d/463.security.md.Two scope notes
docs/observability.md:288is deliberately not keyed: that line passes no--status-file, and the row it can leave is an honestnot run, not the green lie this card is about. Test 6's exact count of 2 is a ratchet — a third documented invocation fails it whether or not it is keyed, which is the intended effect (a new invocation updates the count and passes the key in the same PR).'skipped','not-reached'); the card's sentences are rendering only, andreadRestoreDrillnormalises nothing.