[colbymchenry]

Summary

An indexing run that is stopped partway, by a crash, an out-of-memory kill or the liveness watchdog itself, can leave some of the database's secondary indexes dropped. Every writable open then rebuilt them on the caller's thread. On a large index that took longer than the MCP server's liveness watchdog allows, so the daemon was killed before the index committed, and the same thing happened on every restart. Opens on a watched event loop now rebuild those indexes on a worker thread, and the project is served once they are back.

Root cause

The resolution phase drops 5 unresolved_refs and 5 edges indexes while it runs on 150k or more refs, and the fresh-init parse phase drops more. Since #1562, DatabaseConnection.open() recreates whatever is missing, synchronously, one CREATE INDEX after another. Each statement scans and sorts its whole table. While it sorts it writes nothing to the database or its WAL, so the #1231 disk-progress deferral can't help: the watchdog sees a silent main thread with no disk progress and kills it.

The killed daemon's proxy then re-serves the in-flight request in-process, runs the same rebuild, and is killed too.

Fix

• DatabaseConnection.open({ deferIndexRecovery }) records the missing DDL instead of running it.
  • recoverIndexesOffThread() runs that DDL on a worker thread with its own connection. It is single-flight, rejects on failure so the next call tries again, and leaves the CREATE INDEX statements as before (each commits on its own).
  • The worker uses SQLite's default temp store (TEMP_STORE=1 in node:sqlite), so a big sort spills to disk instead of the heap. The main connection uses MEMORY.
  • The worker runs a full GC and exits by itself, as worker-teardown.ts recommends for Windows.
• CodeGraph.open() (async) defers the rebuild and awaits it before returning the instance. That covers the MCP default project on the daemon, direct and proxy-fallback paths, and the CLI.
• The MCP's synchronous opens use openSync({ deferIndexRecovery: true }) plus a new cg.recoverIndexes(). These are projectPath projects in ToolHandler, and MCPEngine.retryInitializeSync.
  • ToolHandler.execute awaits the rebuild before serving. This wait is not time-boxed like the catch-up gate: without the indexes a read scans whole tables, which would stall the same thread.
  • The project lifecycle awaits it before its catch-up sync writes, so the sync never waits out the rebuild's write lock.
  • The retry path starts the query pool only after the rebuild, so pool workers don't race it for the write lock.
• A synchronous open that asks for nothing still rebuilds in line, as before (CLI openSync, workers).

Validation

• Reachable on current main (b2a8487). I killed a codegraph init of django (79502d0) 1.5 s into its resolution window.

  • It left exactly the field report's set of indexes missing (5 ref + 5 edge, index_state: indexing), in a 157 MB index with 266k refs.
  • The in-line rebuild there is about 750 ms of main-thread work, up to 200 ms per index.

• Real daemon on Windows, scaled. A 3M-ref / 1M-edge index (781 MB) left in the same state, rebuilds of 0.5–3.7 s each, watchdog 1.5 s:

  	daemon starts	killed	indexes back after	reply to the client
  main	4	4	2/10, stuck	none; the launcher's in-process fallback was killed too
  fix	2	0	10/10 after the first	13 s on the first start (about 11 s of rebuild), 1.6 s on the second

  With the fix, the longest main-thread block in a run was 506 ms, and 615 ms in a rerun on the final build.

• New __tests__/index-recovery-off-thread.test.ts, 9 cases. Main's source: 8 fail, 1 passes (the in-line guard). With the fix: 9 pass.

  • Spies record SQL on the server thread. They show no CREATE INDEX there for:
    • CodeGraph.open()
    • a projectPath call, watched and unwatched (the watched case also checks that the rebuild finishes before the catch-up sync starts)
    • the retry path
    • the startup path
  • Every index is back before the call returns.
  • A real liveness watchdog (500 ms, heartbeat only) kills main mid-rebuild of a 300k-ref index and leaves the fix alone.
  • 6 mutations each failed at least one case:
    • in-line rebuild in CodeGraph.open
    • in-line rebuild in projectPath opens
    • in-line rebuild on the retry path
    • no await in execute
    • no await before the lifecycle's sync
    • no single-flight

• Related suites (foundation, query pool, projectPath lifecycle, writer lock, WSL shared index, reopen-on-replace, WAL heal, concurrent locking, unindexed, liveness watchdog): all pass.

• Full suite on this commit (kernel built, CODEGRAPH_KERNEL_EXPECT=1, 4 workers): 720 files, 8,521 passed, 0 failed, 82 skipped. No reruns needed.

Fixes #1887

Credits: the worker-thread recovery comes from @cbeaulieu-gt's #1888, and from the stale draft #1919 (@colbymchenry), which carries #1888. This PR ports that recovery to current main's open paths.

• Current main serves projectPath projects through a synchronous lifecycle (MCP: projectPath queries leave non-default repository indexes without auto-sync #1835), so instead of turning ToolHandler.getCodeGraph async as fix: recover interrupted secondary indexes off-thread #1888 and Prepare CLI extensions release: verified fixes, artifact proof and deferred viewer commands #1919 did, this keeps those opens synchronous and defers only the index rebuild, which execute and the lifecycle await.
• fix: recover interrupted secondary indexes off-thread #1888's reopenIfReplacedAsync is left out. It covers a database replaced on disk by a rebuild in another process, not an interrupted run. That path never fires on Windows (no inode change), and changing it needs POSIX validation I can't do here.

Left out

• The FTS rebuild that healBulkNodeLoad runs at open for a never-closed bulk node load stays in line. A parse-phase kill in a fresh init leaves an empty database (its journal is in memory until the index completes), so that rebuild is instant in practice. The field report's index had all three FTS triggers.
• The memory peaks in the field comment were not measured. The worker's on-disk temp store bounds the sort's memory, but I did not reproduce a 14 GB index.

🤖 Generated with Claude Code