Skip to content

Reduce peak memory during parallel word-index merges - #774

Closed
xus1novv wants to merge 2 commits into
justrach:release/0.2.5862from
xus1novv:fix/release-word-shards-early
Closed

xus1novv wants to merge 2 commits into
justrach:release/0.2.5862from
xus1novv:fix/release-word-shards-early

Conversation

@xus1novv

@xus1novv xus1novv commented Oct 28, 2026 •

Copy link
Copy Markdown

Problem

During a parallel cold scan, all worker arenas remain allocated until every word-index shard has been merged. At the end of the merge, the completed destination index and all source shards coexist in memory.

Release each worker arena immediately after its shard has been copied into the destination index. The existing cleanup path still handles the current and remaining workers when a merge fails. This reduces the merge-phase peak; it does not impose an overall indexing memory limit or establish that it resolves every large-repository OOM.

Evidence

  • In a real incident on 2026-10-28 at 19:40:56 (UTC+05:00), Linux terminated a codedb process after reporting 29,776,092 KiB of anonymous resident memory (approximately 28.4 GiB / 30.5 GB). This is an observed OOM, not evidence that this change alone prevents that incident.
  • Pinned Zig 0.17.0-dev.813+2153f8143, Linux x86_64, ReleaseFast, eight scan workers.
  • A synthetic cold-index fixture with 128 text files of 512 KiB each used 450,564 KiB peak RSS on the base commit and 388,116 KiB with this change (one measured run of each, using /usr/bin/time; processes capped at 1 GiB address space).
  • The new memory-lifetime regression failed on the original merge order (expected 0, found 1586 bytes retained) and passes after the change. A second regression covers merge failure and caller cleanup.
  • zig build test-index -Dtest-filter='watcher: parallel word-index shards match sequential' passes.
  • zig fmt --check src/watcher.zig and git diff --check pass.

Verification limitations

  • zig build test -j1 fails in this environment on two watcher notification tests, repeated across test binaries. The same two tests fail in an unmodified checkout of the base commit.
  • The MCP end-to-end suite passes 75/76 scenarios; the semantic index scenario fails with AnnDirectorySyncFailed on both the base and modified binaries in this environment.

justrach and others added 2 commits October 27, 2026 02:12
…ults

Enable the verified bounded mmap dictionary on current main, preserving the integrated watcher CPU fixes and retrieval breadth. Signed release publication remains gated by the existing multi-platform workflow.
@github-actions

Copy link
Copy Markdown

👋 Thanks for the contribution! Quick heads-up: this repo lands changes on the current release/* branch, not main.

Please retarget this PR via Edit → base branch to the active release/* branch shown in the repository branch list.

(Automated hint — reply here if you need a hand.)

@xus1novv
xus1novv changed the base branch from main to release/0.2.5862 October 28, 2026 18:02
@xus1novv
xus1novv marked this pull request as ready for review October 29, 2026 07:07
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ⚠️ Failed 2026-10-29T07:07:21.189471Z
We couldn’t complete this review. Comment @codex review to try again.
88494ba Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

@xus1novv xus1novv closed this Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants