Support fine-grained SVS index

[ethanglaser]

Adds support for using newly-added separate fine-grained SVS index with minor corresponding test revisions

Includes SVS 0.5.0 rc1 binaries, which will eventually be swapped out for the official release binaries after validation.

Redisearch CI failures at test_vecsim_svs.py:711 and test_vecsim_svs.py:729 come from changes in internal logic not being reflected in tests yet.

1. Each deletion now submit a single label consolidation job, that can change size of the reverse edges lists. This breaks the test_vecsim_svs.py:711.
2. Some of the submitted consolidation jobs can be unfinished before GC launch. This breaks the test_vecsim_svs.py:729.

---

Note

High Risk
Large changes to tiered SVS ingestion, delete/consolidate concurrency, and locking around a new concurrent index dependency affect correctness of search and updates under load.

Overview
This PR moves VecSim onto SVS 0.5.0 rc2 prebuilt binaries and switches the backend from static Vamana types to svs::concurrent mutable indexes with segmented blocked storage/graph allocators so readers can query while writers append.

SVSIndex now keeps the implementation in a shared_ptr guarded by mutexes, can stay empty until the first insert (ready()), and routes adds/deletes/consolidation/compact through that concurrent API—including replace_external_id when the new headers provide it. Tiered SVS is reworked: instead of periodic batch snapshots with swap journals, it buffers training in the flat tier, initializes the backend once, then ingests remaining vectors via per-label insert jobs (with defer/retry while init runs) and schedules consolidate jobs after deletes so soft-deleted labels don’t block relabels. Query paths rely less on mainIndexGuard because the concurrent backend serializes internally.

Shared tiered plumbing (TieredInsertJob, invalid-job tracking, flat removal after ingest, indexSize/getDistanceFrom locking hooks via new ScopedLocks) is lifted into VecSimTieredIndex, with tiered HNSW adapted to match. Tests and size estimates account for reverse-edge overhead; Python runGC calls the index directly so non-tiered SVS compacts too.

^{Reviewed by Cursor Bugbot for commit 9e602e9. Bugbot is set up for automated code reviews on this repo. Configure here.}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 4 committers have signed the CLA.

✅ ofiryanai
❌ Dmitry Razdoburdin
❌ razdoburdin
❌ ahuber21

---

Dmitry Razdoburdin seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 4 potential issues.

^{Reviewed by Cursor Bugbot for commit 9e602e9. Configure here.}

[cursor]

Full-buffer overwrite skips pending jobs

High Severity

When the frontend is at flat_buffer_bound, addVector writes straight to the SVS backend and only soft-deletes that backend copy. It never invalidates a pending insert job or removes the label from the frontend. A later ingest can put the old blob back (and addVectorsImpl can drop the new one), and the call returns 1 for an overwrite instead of 0.

 

^{Reviewed by Cursor Bugbot for commit 9e602e9. Configure here.}

[cursor]

Init overwrite reports a new insert

Medium Severity

During the training/init buffer, a single-value overwrite calls deleteAndUpdateInitIds and then frontendIndex->addVector, which treats the write as a new row and returns 1. Callers that use 0 vs 1 to distinguish update from insert will count an extra vector until the backend is initialized.

Additional Locations (1)

• src/VecSim/algorithms/svs/svs_tiered.h#L927-L933

 

^{Reviewed by Cursor Bugbot for commit 9e602e9. Configure here.}

[cursor]

Empty index skips compression retraining

High Severity

backendInitSubmited is set once when the first training batch is scheduled and never cleared. After the last backend vector is deleted and impl_ is reset, later async adds skip the training buffer and let a single insert job initImpl the compressed index from one point.

Additional Locations (2)

• src/VecSim/algorithms/svs/svs_tiered.h#L268-L269
• src/VecSim/algorithms/svs/svs_tiered.h#L980-L982

 

^{Reviewed by Cursor Bugbot for commit 9e602e9. Configure here.}

[cursor]

GC compacts without consolidating deletes

Medium Severity

runGC now only calls compact() and zeroes num_marked_deleted. Per-label consolidate jobs are not drained first, so compact can run while deleted graph edges are still unpatched. SVS documents compact as the step that follows consolidate.

Additional Locations (1)

• src/VecSim/algorithms/svs/svs_tiered.h#L604-L628

 

^{Reviewed by Cursor Bugbot for commit 9e602e9. Configure here.}