Repository navigation
fix: recover interrupted secondary indexes off-thread - #1888
Closed
cbeaulieu-gt wants to merge 1 commit into
Closed
cbeaulieu-gt wants to merge 1 commit into
cbeaulieu-gt wants to merge 1 commit into
Conversation
Contributor
Author
|
I noticed that maintainer draft #1919 carries the #1887 recovery fix as commit Please let me know whether you would prefer #1888 rebased as the focused integration path, or whether #1919 is intended to carry the fix forward. I’m happy to update this branch if the standalone PR is still useful.
|
Closed
7 tasks
colbymchenry
added a commit
that referenced
this pull request
Oct 22, 2026
…led at every start (#2794) Root cause: a run killed inside a bulk-load window (a crash, an OOM kill, the liveness watchdog itself) leaves that window's secondary indexes dropped, and since PR #1562 every writable open rebuilds them in DatabaseConnection.open(), one CREATE INDEX after another on the caller's thread. Each one scans and sorts its whole table. On a large index the sort writes nothing to the database or its WAL, so on an MCP server the liveness watchdog saw a silent main thread with no disk progress, and killed the process before the index committed. Each restart died at the same index, and the proxy's in-process fallback died the same way on the request it re-served. Fix: an open on a watched event loop leaves the rebuild to a worker thread with its own connection, and the project is served once the rebuild is done. - DatabaseConnection.open({ deferIndexRecovery }) records the missing DDL instead of running it; recoverIndexesOffThread() runs it on a worker (single-flight, rejects on failure so the next call retries; SQLite's default temp store, so the sort spills to disk; the worker collects its garbage and exits by itself, per worker-teardown.ts). - CodeGraph.open() (the MCP default project, the CLI) defers and awaits it. - openSync({ deferIndexRecovery }) plus cg.recoverIndexes() for the MCP's synchronous opens: the ToolHandler's projectPath projects and the engine's retry path. ToolHandler.execute awaits the rebuild before serving (not time-boxed: reads without the indexes scan whole tables), the project lifecycle awaits it before its catch-up sync writes, and the retry path starts the query pool after it. - Synchronous opens that ask for nothing keep rebuilding in line, as before. Validation: - Reproduced on main b2a8487: a django 79502d0 init killed 1.5s into the resolution window left the field report's index set missing (5 ref + 5 edge indexes, index_state indexing); the in-line rebuild is about 750ms of main-thread work there. - Real daemon on Windows, a 3M-ref / 1M-edge index left the same way (781 MB), watchdog 1.5s: main was killed in 4 of 4 starts, stuck at 2 of 10 indexes, the launcher's in-process fallback killed too, no reply. The fix rebuilt all 10 in about 11s with no main-thread block over 506ms, answered the call, and the next start opened in about 120ms. - New __tests__/index-recovery-off-thread.test.ts (9 cases): main's source 8 fail / 1 pass (the in-line guard), fix 9 pass. Spies show no CREATE INDEX on the server thread for CodeGraph.open, a projectPath call (watched and not), the retry path and the startup path; a real watchdog (500ms) kills main mid-rebuild of a 300k-ref index and leaves the fix alone. 6 mutations each failed at least one case. - Contributor PR #1888 and the stale draft PR #1919 were read for guidance; this ports their worker recovery to current main's open paths. Fixes #1887 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner
|
Thank you, your analysis and this PR guided the fix. #2794 is merged on main: after an interrupted run, the index rebuild now happens on a worker thread, so the watchdog no longer kills the server at every start. It ships in the next release (after 1.6.2) with credit to you, so we'll close this one. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why
Crash recovery previously executed every missing
CREATE INDEXsynchronously insideDatabaseConnection.open(). On a large graph, one index build can block the MCP event loop long enough for the liveness watchdog to kill the process before SQLite commits any visible file progress, so the next restart repeats the same work and can be killed again.The synchronous API remains available for compatibility. Async callers now use the off-thread recovery path and receive the connection only after every required index has been restored.
Verification
npm run buildnpx vitest run __tests__/foundation.test.tsnpx vitest run __tests__/liveness-watchdog.test.tsnpx vitest run __tests__/concurrent-locking.test.tsnpx vitest run __tests__/mcp-unindexed.test.tsnpx vitest run __tests__/db-reopen-on-replace.test.tsThe watchdog regression creates a 600,000-node interrupted database under a scaled timeout. The former synchronous open is killed by the watchdog; the async recovery completes and restores the full index set.
Fixes #1887