Skip to content

perf(scan): make the Codex model backfill touch only NULL-model rows - #19

Merged
luwill merged 1 commit into
luwill:mainfrom
amenjason936:perf/idle-scan-backfill
Oct 20, 2026
Merged

luwill merged 1 commit into
luwill:mainfrom
amenjason936:perf/idle-scan-backfill

Conversation

@amenjason936

Copy link
Copy Markdown
Contributor

Related to #13. Independent of #16/#17/#18. The only overlap is the one-line test registration in test/run.mjs.

Problem

Every scan, including idle ones where no file changed, ended with _inheritCodexModels(). That function runs one backfill UPDATE per Codex session:

UPDATE events SET model = ? WHERE tool = 'codex' AND session_id = ? AND model IS NULL

The planner serves this through idx_events_session (session_id=?). So for each session, every one of its events is visited just to confirm that model is not NULL. Since there are no NULL rows in the steady state, all of this work is wasted. With 1M Codex events across 5000 sessions, that is about 0.5 s per scan, and the dashboard runs a scan roughly every second while an agent is active.

Fix

  • Partial index: CREATE INDEX IF NOT EXISTS idx_events_model_null ON events(session_id) WHERE model IS NULL, added in SCHEMA so existing databases get it on next start. It holds only rows that are missing a model, which is usually none, so each probe is O(log n) on an almost-empty index. Building it on a 1M-row database took ~100 ms, once. The UPDATE pins it with INDEXED BY, because on a database without sqlite_stat1 the planner sometimes still picks idx_events_session; I saw this on a fresh database.
  • Skip scans with no Codex change: a model becomes known only when a Codex file is re-parsed, either from its own records or because its parent's file was parsed. So _inheritCodexModels now runs only on scans that sent a Codex file to the collector. Each Scanner instance still runs it once on its first scan, which covers NULLs left by older versions or by an interrupted run.

Numbers

Setup: 1M Codex events, 5000 rollout files, Linux, Node 22.22. Values are averages over 10 scans; CPU is from process.cpuUsage() and includes libuv threadpool stats.

before after
idle rescan (no file changed) 1.37 s CPU, backfill 525 ms 0.49 s CPU, backfill skipped
Codex active (one file appended per scan) 0.94 s CPU, backfill 459 ms 0.47 s CPU, backfill 44 ms

The remaining ~0.5 s is the directory walk plus one stat and one files lookup per file. Reducing that further means scanning only the paths that fs.watch reports, which is a bigger change. I've left it out of this PR.

Tests

test/idle-scan.mjs tests behaviour, not timings:

  • Query plan: it captures the UPDATE statement the scanner actually prepares and asserts that EXPLAIN QUERY PLAN uses idx_events_model_null.
  • Late parent model: the child (a resume link) is scanned while its parent has no model yet, so its events stay NULL. The parent's model then appears in a later scan, and the child inherits it.
  • Backfill run counts:
    • First scan: runs.
    • Idle rescans: do not run.
    • A scan after a Codex file changed: runs.
    • A new Scanner on the same database with nothing changed: runs once and restores models that were set back to NULL by hand.

Without the source changes, 4 of these 5 checks fail. Full npm test is green.

🤖 Generated with Claude Code

Every scan ran one backfill UPDATE per Codex session. Through
idx_events_session that visits every event of the session, so with 1M
Codex events across 5000 sessions it cost ~0.5 s per scan, even when
nothing had changed.

- Add a partial index on events(session_id) WHERE model IS NULL and pin
  the UPDATE to it with INDEXED BY. The index holds only rows that are
  missing a model, which is usually none. Backfill: 460 ms -> 44 ms.
- Skip _inheritCodexModels on scans that re-parsed no Codex file. A
  model only becomes known when the file, or its parent, is parsed.
  Each Scanner instance still runs it once to cover NULLs left behind by
  older versions.

Idle rescan at 1M events / 5000 Codex files: ~1.3 s -> ~0.5 s CPU.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@luwill

luwill commented Oct 20, 2026

Copy link
Copy Markdown
Owner

Reviewed and verified in a clean worktree — LGTM.

  • Plans: on a fresh database (no sqlite_stat1) the planner picks idx_events_session (session_id=?) even though the partial index exists, which confirms the INDEXED BY pin is necessary, not just cosmetic. The pinned form plans as SEARCH events USING INDEX idx_events_model_null (session_id=?).
  • Row equivalence: ran both variants side by side on identical fresh databases (codex NULL + codex set + other-session NULL + a non-codex NULL row). Both update exactly the same single row and leave everything else untouched — changes and resulting table state identical.
  • Skip logic: agree with the reasoning — model knowledge can only change when a Codex file is re-parsed, so unchanged rounds have nothing to backfill, and the per-instance first scan covers legacy/interrupted NULLs. CLI invocations get a fresh Scanner and thus always backfill once, which keeps scan/doctor self-healing.
  • Full npm test green locally; CI green.

One correction to the description: this does not only overlap #16–#18 on the test/run.mjs line — #20 and this both edit src/scanner.js (constructor fields and the post-finally region), and a trial merge shows a real conflict in both files. Whichever of #19/#20 merges second needs a small scanner.js rebase, not just the one-liner.

@luwill
luwill merged commit 9f8dc38 into luwill:main Oct 20, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants