Skip to content

perf(index): stop tracking the files being indexed - #687

Merged
daveshanley merged 1 commit into
pb33f:mainfrom
Tjmstudios:drop-indexing-set
Nov 1, 2026
Merged

daveshanley merged 1 commit into
pb33f:mainfrom
Tjmstudios:drop-indexing-set

Conversation

@Tjmstudios

@Tjmstudios Tjmstudios commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

AddIndexingFile copies the whole set of files being indexed on every call, so the cost is quadratic in how deeply external files nest, and indexRemoteFile calls it three times per file. Nothing in the library has read that set since #669 removed the lookup in find_component_external.go, so this stops building it.

I kept AddIndexingFile, GetIndexingFiles, IsFileBeingIndexed and IndexingFilesKey since they're public, and marked them deprecated. Let me know if you'd rather remove them.

BenchmarkRolodex_NestedRemoteFiles indexes a chain of remote files where each one references the next. Median of 6 interleaved runs:

Depth main this PR
10 0.46ms, 291KB, 3,410 allocs 0.41ms, 264KB, 3,226 allocs
100 7.2ms, 5.06MB, 33,674 allocs 3.8ms, 2.47MB, 29,928 allocs
500 79ms, 76.8MB, 157,745 allocs 19ms, 10.95MB, 131,610 allocs

Shallow chains barely change.

Anything outside the library that reads GetIndexingFiles or IsFileBeingIndexed during indexing will no longer see the library's files.

The comment I removed from rolodex_file_loader.go said the set prevents deadlocks on circular references. I don't think that applies since #669, but please check. I added a test that circular remote files finish indexing, which also passes on main.

Also added tests for a nested remote chain and for AddIndexingFile, which would otherwise have no coverage.

go test ./... and go test -race ./... pass with LIBOPENAPI_RUN_NETWORK_TESTS=1, except FuzzAppendJSONString/seed#5 in json, which fails on main too (Go 1.27.0).

AddIndexingFile copies the whole set on every call, which is quadratic in how deeply external files nest. Nothing in the library has read the set since the lookup in find_component_external.go was removed, so stop building it and deprecate the exported helpers.
@codecov

codecov Bot commented Nov 1, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (ae1153a) to head (825d854).

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #687   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files          313       313           
  Lines        39557     39552    -5     
=========================================
- Hits         39557     39552    -5     
Flag Coverage Δ
unittests 100.00% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@daveshanley

Copy link
Copy Markdown
Member

Thanks!

@daveshanley
daveshanley merged commit 0a89094 into pb33f:main Nov 1, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants