Repository navigation
Storing is extremely slow for large repos #26
Description
I've updated to v1.19.5 and have run the indexing over my repository. I've found that as the index gets larger and larger, the store phase starts taking a huge amount of time. At the beginning it was taking a couple of seconds for 50 chunks, and now it's taking 10+ minutes for 50 chunks. Is there any way to accelerate this?
For instance: Store phase: 13 files stored in 1391.8s (running total 17090 chunks)
My data folder is 6+ GB at the moment:
[5.7G] data
├── [5.6G] chunks.lance
│ ├── [219M] data
│ ├── [174K] _deletions
│ ├── [ 12M] _transactions
│ └── [5.4G] _versions
├── [ 38] index.lock
├── [ 46M] keyword-index.json
├── [ 13K] __manifest
│ ├── [4.4K] _transactions
│ └── [4.8K] _versions
├── [1.0M] manifest.json
└── [ 28M] opencode-rag.log
Also, with the new change to interleave the stages, it would be nice if some of them could be run in parallel. For instance, the storing stage doesn't use the GPU, so it could theoretically compute the next stage that uses the GPU at the same time.
Activity
MrDoe commentedon Aug 6, 2026
Thanks for reporting this! I'll take a look into this and see what I can do to improve this. Maybe that's also a limitation of LanceDB. I have another project with Postgres/pgai and another with ClickHouse and they both scale really well on large datasets. But I chose LanceDB intentionally here, because it is so lightweight and compact. Maybe it's getting on its limits here.
I can try this on a separate branch and do some benchmarking, but this takes some time to implement (even with the help of AI).
But your idea to run the store process in parallel is a good one that I'll definitely try out next.
MrDoe commentedon Aug 6, 2026
Additionally, try to disable logging by setting it to "none" to generally improve indexing speed.
MrDoe commentedon Aug 7, 2026
I optimized the indexing workflow now and made some other performance improvements. Can you please try it with the latest version and give me some feedback if it is faster now?