Skip to content

fix(hnsw): handle invalid keys in group-by query with filter (#781) - #782

Open
thlurte wants to merge 1 commit into
alibaba:mainfrom
thlurte:fix-hnsw-group-by-invalid-key
Open

thlurte wants to merge 1 commit into
alibaba:mainfrom
thlurte:fix-hnsw-group-by-invalid-key

Conversation

@thlurte

@thlurte thlurte commented Sep 24, 2026

Copy link
Copy Markdown

Problem

Fixes #781.

When executing Collection.group_by_query with a scalar filter predicate on an HNSW index, the query returned empty results with error logs:

ID is out of range: id[18446744073709551615] count[...]
Fetch group by field failed: field[...] row_id[18446744073709551615]

Root Cause:

  1. During HNSW group neighbor expansion (expand_neighbors_by_group), search heap nodes may return kInvalidKey (UINT64_MAX) when unmapped or deleted.
  2. Passing UINT64_MAX to DocFilter::is_filtered() caused an out-of-bounds access in the bitmap and returned nullopt (unfiltered).
  3. The unfiltered invalid key was then passed to VectorRecallNode::group_fun, which invoked segment_->fetch with row_id = UINT64_MAX (-1), failing to fetch the group field and aborting group formation.

Solution

  1. HNSW Algorithms (hnsw_algorithm.cc, hnsw_sparse_algorithm.cc, hnsw_rabitq_query_algorithm.cc):
    • Skip kInvalidKey during initial group partitioning.
    • Guard candidate queue refill against kInvalidKey.
    • Update filter lambda to treat kInvalidKey as filtered (true), preventing unmapped nodes from being selected or expanded.
  2. DocFilter (doc_filter.cc):
    • Add boundary check if (id == std::numeric_limits<uint64_t>::max() || id >= segment_->doc_count()) return true; to prevent bitmap index overruns.
  3. VectorRecallNode (vector_recall_node.cc):
    • Guard group_fun against invalid row_id / UINT64_MAX before calling segment_->fetch().
  4. Tests (collection_test.cc):
    • Added regression test Feature_GroupByQuery_HnswWithFilter verifying filtered group_by_query on HNSW index for both in-memory and mmap collections.

…#781)

- Guard against kInvalidKey (UINT64_MAX) during HNSW, Sparse HNSW, and RaBitQ HNSW group neighbor expansion.

- Prevent ID out of range errors in DocFilter and VectorRecallNode when querying invalid/unmapped doc keys.

- Add regression test Feature_GroupByQuery_HnswWithFilter in collection_test.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: Collection.group_by_query returns empty results with ID-out-of-range errors (UINT64_MAX)

4 participants