Repository navigation
perf(scan): make the Codex model backfill touch only NULL-model rows - #19
Merged
Merged
Conversation
Every scan ran one backfill UPDATE per Codex session. Through idx_events_session that visits every event of the session, so with 1M Codex events across 5000 sessions it cost ~0.5 s per scan, even when nothing had changed. - Add a partial index on events(session_id) WHERE model IS NULL and pin the UPDATE to it with INDEXED BY. The index holds only rows that are missing a model, which is usually none. Backfill: 460 ms -> 44 ms. - Skip _inheritCodexModels on scans that re-parsed no Codex file. A model only becomes known when the file, or its parent, is parsed. Each Scanner instance still runs it once to cover NULLs left behind by older versions. Idle rescan at 1M events / 5000 Codex files: ~1.3 s -> ~0.5 s CPU. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 20, 2026
Owner
|
Reviewed and verified in a clean worktree — LGTM.
One correction to the description: this does not only overlap #16–#18 on the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related to #13. Independent of #16/#17/#18. The only overlap is the one-line test registration in
test/run.mjs.Problem
Every scan, including idle ones where no file changed, ended with
_inheritCodexModels(). That function runs one backfill UPDATE per Codex session:The planner serves this through
idx_events_session (session_id=?). So for each session, every one of its events is visited just to confirm thatmodelis not NULL. Since there are no NULL rows in the steady state, all of this work is wasted. With 1M Codex events across 5000 sessions, that is about 0.5 s per scan, and the dashboard runs a scan roughly every second while an agent is active.Fix
CREATE INDEX IF NOT EXISTS idx_events_model_null ON events(session_id) WHERE model IS NULL, added inSCHEMAso existing databases get it on next start. It holds only rows that are missing a model, which is usually none, so each probe is O(log n) on an almost-empty index. Building it on a 1M-row database took ~100 ms, once. The UPDATE pins it withINDEXED BY, because on a database withoutsqlite_stat1the planner sometimes still picksidx_events_session; I saw this on a fresh database._inheritCodexModelsnow runs only on scans that sent a Codex file to the collector. EachScannerinstance still runs it once on its first scan, which covers NULLs left by older versions or by an interrupted run.Numbers
Setup: 1M Codex events, 5000 rollout files, Linux, Node 22.22. Values are averages over 10 scans; CPU is from
process.cpuUsage()and includes libuv threadpoolstats.The remaining ~0.5 s is the directory walk plus one
statand onefileslookup per file. Reducing that further means scanning only the paths thatfs.watchreports, which is a bigger change. I've left it out of this PR.Tests
test/idle-scan.mjstests behaviour, not timings:EXPLAIN QUERY PLANusesidx_events_model_null.Scanneron the same database with nothing changed: runs once and restores models that were set back to NULL by hand.Without the source changes, 4 of these 5 checks fail. Full
npm testis green.🤖 Generated with Claude Code