[Bug](lance) Vector search misses a later same-column index matching the query metric

[FANNG1]

Problem

Doris may fall back to Flat Search even when a metric-compatible Lance vector index exists on the queried column.

Lance permits multiple named logical vector indexes on the same vector column. For example:

vec_l2      column=vector, metric=L2
vec_cosine  column=vector, metric=COSINE

For a vector_search() request with metric=cosine, Doris should be able to select vec_cosine. Currently, if vec_l2 appears first in the index metadata returned to the FE, Doris selects all segments belonging to vec_l2, checks their metric afterwards, and falls back to fragment-based search when the metric does not match. It does not continue looking for vec_cosine.

The query result remains correct because Lance executes Flat Search with the requested metric, but the available ANN index is silently missed. Query latency and resource usage can therefore depend on index metadata ordering.

Current implementation

LanceScanNode.selectIndexSegments() first locks onto the first logical index covering the vector field:

if (selectedIndexName == null) {
    selectedIndexName = segment.getIndexName();
}
if (selectedIndexName.equals(segment.getIndexName())) {
    selectedSegments.add(segment);
}

Only after that selection does createIndexSegmentSplits() call metricMatches():

List<LanceIndexSegmentInfo> matchingSegments = selectIndexSegments(
        metadata.getIndexSegments(), vectorFieldId);
if (matchingSegments.isEmpty() || !metricMatches(vectorSearchParam, matchingSegments)) {
    return Optional.empty();
}

Relevant code:

• fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanNode.java
• fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceMetadataLoader.java

The FE already loads the information needed to make a metric-aware decision:

• logical index name
• indexed field IDs
• physical segment UUIDs
• index metric
• fragment coverage

This is not a Lance index-format limitation. Doris can select a logical index by sending that index's physical segment UUIDs to the BE; no new Lance index_name scanner parameter is required for automatic metric-aware selection.

Expected behavior

When use_index=true, Doris should:

1. Group physical index segments by logical index name.
2. Keep logical indexes covering the requested vector field.
3. Resolve the requested metric using Doris semantics (L2 when the query metric is omitted or DEFAULT).
4. Select a logical index whose segments use the requested metric.
5. Route only that logical index's segment UUIDs to the BE.
6. Fall back to Flat Search only when no compatible logical index can be planned safely.

If more than one logical index matches both the field and metric, Doris should either define a deterministic documented policy or later expose an optional index_name query property. Explicit user selection is a separate enhancement; it is not required to fix the missed compatible-index case.

Legacy index metadata without a metric or fragment bitmap should continue to use the existing conservative fallback unless a safe compatibility path is defined.

Reproduction

1. Create a Lance dataset with one vector column.
2. Create two named logical indexes on that column, one using L2 and one using cosine, ensuring the L2 index is returned first by describeIndices().
3. Run Doris vector_search() with metric=cosine and use_index=true.
4. Inspect EXPLAIN.

Current behavior:

lanceSearchIndexSegments=0

and the query falls back to fragment search, even though the cosine index exists.

Expected behavior:

lanceSearchIndexSegments>0

with only the vec_cosine physical segments assigned to indexed splits.

Acceptance criteria

• FE unit test: L2 index appears first, cosine index second, and a cosine query selects the cosine segments.
• FE unit test: reversing metadata order does not change the selected metric-compatible logical index.
• FE unit test: segments from different logical index names are never mixed in one index plan.
• FE unit test: no compatible metric still produces the existing Flat Search fallback.
• FE unit test: omitted/DEFAULT metric follows the documented Doris L2 default.
• Regression test proves physical indexed execution, using both EXPLAIN segment planning and an nprobes/ef discriminator rather than only comparing indexed and flat results.
• Documentation no longer recommends one index per vector column solely because Doris cannot find a later metric-compatible index.

Related

• [Tracking] Lance feature matrix for Doris 4.2 #66340
• [test](lance) Add indexed coverage for cosine, dot and non-Float32 element types #67039 currently uses one vector column per matrix cell to avoid this selection limitation; it expands coverage but does not change the production selection logic.

[Doris-Breakwater]

Breakwater-GitHub-Analysis-Slot: slot_9e82f2ff694b

Initial triage

Verdict: confirmed FE index-planning bug. The result remains correct through the fragment/Flat Search fallback, but index use—and therefore latency and resource consumption—incorrectly depends on describeIndices() ordering.

I verified the report against the current upstream/branch-4.1 implementation at 3bfc343de0c496597a59a8377ff2043339987ba8. The issue currently has no labels, assignee, milestone, or prior comments. Suggested existing labels are area/catalog, kind/performance, and kind/need-regression-test.

Verified facts

• LanceMetadataLoader.loadVectorIndexSegments() preserves every physical segment's logical index name, field IDs, UUID, fragment bitmap, and the normalized metric_type from its IndexDescription. The FE therefore already has enough information to choose among same-column logical indexes.
• LanceScanNode.selectIndexSegments() sets selectedIndexName from the first segment covering vectorFieldId and then retains only segments with that name. createIndexSegmentSplits() calls metricMatches() only after this irreversible selection. Thus an earlier L2 index prevents a later cosine index from being considered for a cosine query.
• When that metric check fails, createIndexSegmentSplits() returns empty and getSplits() creates fragment splits; the index counters remain zero. This matches the reported EXPLAIN behavior.
• The BE path already serializes TLanceFileDesc.index_segment_uuids into lance_scanner_set_index_segments(). Selection by logical index_name is not required for this fix: routing only the chosen logical index's physical UUIDs is sufficient.
• The requested-metric resolver already treats an omitted metric and TVectorMetric.DEFAULT as L2. Missing metric metadata and missing fragment bitmaps already cause conservative fallback.
• Existing FE coverage tests one logical index with multiple physical segments and a single-index metric mismatch, but it does not cover multiple logical indexes on the same field. The open test PR [test](lance) Add indexed coverage for cosine, dot and non-Float32 element types #67039 also explicitly uses one vector column per matrix cell because of this limitation, independently corroborating the gap.

No runtime profile or additional user logs are needed to establish this root cause; the control flow is deterministic. I did not run an end-to-end Lance reproduction, so physical execution still needs to be proven by the regression test requested below.

Recommended implementation

1. Resolve the requested metric once using the existing Doris semantics.
2. Group target-field segments by logical index name before selecting a candidate.
3. Evaluate each complete group for metric compatibility and safe plannability. If a group has an unknown metric, a missing fragment bitmap, or produces no visible segment split, continue to the next candidate instead of allowing that group to block a later safe match.
4. Choose among multiple safe field+metric matches with an order-independent policy. A minimal policy is lexicographically smallest logical index name; if maintainers prefer best coverage, compare visible covered fragments/rows first and use the name as a stable tie-breaker. Whichever rule is chosen should be documented.
5. Pass only that group's physical UUIDs to the existing split planner and retain the existing Flat Search splits for visible fragments not covered by the selected index.
6. Preserve whole-query conservative fallback when no logical group can be planned safely. No BE or Thrift API change appears necessary.

The important detail is to choose a safe plan, not merely the first metric-compatible group: otherwise a compatible legacy group with no fragment bitmap could still hide a later fully usable group.

Validation expected in the fix

• FE unit test: [L2 index, cosine index] with a cosine query selects only the cosine UUIDs.
• Repeat with reversed metadata order and assert the same selected UUIDs/splits.
• Two same-field, same-metric logical indexes: assert the documented deterministic winner and prove UUIDs from different names are never mixed.
• No compatible metric: retain fragment splits and zero index-segment planning.
• Test both an unset metric and explicit DEFAULT; each must select the L2 group.
• Legacy/unsafe candidate tests: an unknown metric or missing fragment bitmap falls back when alone, but does not block a later safe compatible logical index.
• Add a dedicated regression fixture with two named indexes on one vector column. Assert non-zero indexed segment planning in EXPLAIN, then use an nprobes or ef discriminator that demonstrably differs from Flat Search; indexed-vs-flat result equality alone cannot prove physical index use.
• Coordinate the documentation change with [doc](lance) Document the vector element type and metric compatibility matrix doris-website#4082: remove the "one vector index per column" workaround once the fix is available, while retaining the metric-default and legacy-fallback documentation.

Maintainer decisions still needed

• Target branch and any backport scope are not stated in the issue.
• Select and document the tie-breaker for multiple safe indexes with the same field and metric.
• Decide whether the dedicated multi-index regression fixture belongs in this fix or in the currently open matrix-test work, while keeping the FE unit coverage in the fix itself.

[zhangstar333]

hi @FANNG1
This is indeed a bug — i didn't consider all the cases at the time. Definitely worth fixing.

[FANNG1]

hi @FANNG1 This is indeed a bug — i didn't consider all the cases at the time. Definitely worth fixing.

Could you assign this to me? I will fix it.