[solotzg]

What problem does this PR solve?

Issue Number: ref #10233

Related: #11111 (base implementation), pingcap/tidb#71839 (pushdown), pingcap/tidb#71397 (v8.5 backport).

Large STANDARD Boolean queries repeatedly compare document tokens with unmatched terms.

What is changed and how it works?

Functions: route large Local MATCH exact-term queries by collation key

• Lazily map collation keys to all clause/term references, preserving Boolean/equal-key semantics; process each matched bucket once per row across MATCH columns.
• Enable routing for supported exact queries with ≥100 effective term associations (duplicates count), ≥32 document-column tokens, and a conservative 4 MiB allocation estimate.
• Retain linear matching for small queries, NGRAM, prefix, phrase/verification, unsupported collations, oversized tokens (>4,096 bytes), and over-budget queries.

Arena keys and block-local scratch are memory-tracked. SQL/protocol, FULLTEXT metadata, feature gates, and allocator defaults are unchanged.

Check List

• 47 local tests: all 11 admitted collation IDs, compare/key equivalence, malformed UTF-8 tokenization, Boolean/NULL/multi-column semantics, routing and block reuse.
• 28 manual SQL cases: full sorted IDs match TiDB fallback and the original baseline; MPP MATCH plans verified.

Performance

100 absent required terms; 10,000 rows × 1,000 tokens, synthetic ASCII. Baseline 608fb566c3; TiDB 7b255e057f; PD/TiKV v8.5.8. Linux Release (RelWithDebInfo), clang 17, four physical cores, one query thread.

Collation	Baseline median(s), s	Candidate median(s), s	Speedup
utf8mb4_general_ci	14.943	0.595	25.11x
utf8mb4_unicode_ci	19.974	0.863	23.16x
utf8mb4_0900_ai_ci	15.521	0.858	18.08x
utf8mb4_bin (PAD)	13.486 / 13.370	0.354 / 0.353	37.98x
utf8mb4_0900_bin (NO PAD)	12.639 / 12.678	0.365 / 0.361	34.87x

CI rows: candidate 93a1295995, one pair, three samples per version. Binary rows: production source 45db0e5d99 (matching merged production files), two AB/BA pairs, five samples per version per epoch. Medians are listed by pair; speedup is the inverse geometric mean of paired C/B ratios. Separate rounds are not pooled.

Limits

• Early hits can cost more: binary OR about +2.2%; Unicode CI repeated MUST +6.98% / +2.25% in two pairs.
• PAD four-term control: +13.04% / −3.20%. Three independent follow-up pairs, 21 samples per version per epoch, did not reproduce a slowdown beyond 5% or 0.2 ms. Original samples remain; the single-pair cause is unresolved.
• Synthetic single-client results do not establish a production-optimal 100-term threshold or mixed-concurrency/tail guarantees. No real corpus was available.
• Sustained-load RSS increases. Deployment memory-budget acceptance and long-term leak freedom remain unverified.

Follow-up TODOs (separate PRs)

• Investigate an equivalent ASCII fast path for UCA sort-key generation.
• Investigate incremental tokenization/matching where Boolean semantics allow early termination.

Release note

Improve Local MATCH AGAINST performance for large STANDARD exact-term Boolean queries.

Summary by CodeRabbit

• Performance
  • Boolean text matching can use faster exact-term lookups for eligible larger searches, reducing repeated comparisons.
  • The optimized route applies to supported standard queries within its resource limits. Searches that do not qualify continue to use the existing matching behavior, preserving results across both routes.
• Tests
  • Expanded coverage verifies consistent results for optimized and fallback matching, including supported collations, repeated terms, nullable or reused data blocks, and concurrent executions.

[ti-chi-bot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

• Last reviewed commit: 71d093c
• Latest commit, not reviewed: 7599cd6

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• Resume reviews
• Trigger review

No actionable comments were generated in the recent review.

Walkthrough

Local MATCH adds collation-key routing for eligible Boolean queries and retains linear matching for other cases. The change adds a test-facing matcher factory, work counters, and routing tests. The TiDB failpoint image configuration now uses a dedicated environment variable.

Changes: Local MATCH exact-term routing

Layer / File(s)	Summary
Exact-term route construction dbms/src/Functions/FunctionsLocalMatchAgainst.cpp	Eligible query terms are grouped by collation sort key with their clause-term references. Route construction checks query restrictions and an estimated memory budget.
Indexed and linear matching execution dbms/src/Functions/FunctionsLocalMatchAgainst.cpp, dbms/src/Functions/FunctionsLocalMatchAgainst.h	Execution selects indexed or linear matching based on matcher settings and eligibility. Shared clause-state helpers support both paths. The test-facing API exposes matcher modes, work counters, budget, and generation seed.
Routing and matching validation dbms/src/Functions/tests/gtest_local_match_against.cpp	Tests cover result equivalence, route thresholds and fallbacks, collation-key equality, reused and nullable blocks, oversized tokens, and concurrent execution.

TiDB failpoint image configuration

Layer / File(s)	Summary
Failpoint image selection tests/docker/cluster_tidb_fail_point.yaml	The tidb0 image uses TIDB_FAILPOINT_IMAGE instead of deriving the failpoint image from TIDB_IMAGE.

Estimated code review effort: 4 (Complex) | ~60 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant LocalMatch as LocalMatchAgainst
  participant Collator
  participant ExactTermRoute
  participant ClauseState
  LocalMatch->>Collator: Generate a sort key for an analyzed token
  Collator-->>LocalMatch: Return the token sort key
  LocalMatch->>ExactTermRoute: Look up the matching bucket
  ExactTermRoute-->>LocalMatch: Return clause-term references
  LocalMatch->>ClauseState: Update matching clauses

Loading

Possibly related PRs

• pingcap/tiflash#11152: Changes collator sort-key generation and UTF-8 decoding used by the indexed route.
• pingcap/tiflash#11153: Backports the collator sort-key and UTF-8 decoding changes used by the indexed route.
• pingcap/tiflash#11135: Includes collator input-length checks used by the indexed route’s sort-key path.

Suggested reviewers: xzhangxian1008, jayson-huang

Merge Risk: Minimal · up to 71d09

Large exact-term MATCH queries gain an indexed path. Failpoint Compose users must now set TIDB_FAILPOINT_IMAGE.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 13.64% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta…	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Title check		The title names the main change: optimizing large Local MATCH exact-term queries. The changeset adds an exact-term hash/collation-key routing path for large STANDARD Boolean queries, which matches the…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Description check		The description follows the repository template. It explains the problem, implementation, tests, performance evidence, side effects, documentation impact, and release note. It also identifies pending …

Full details: Title check

Explanation

The title names the main change: optimizing large Local MATCH exact-term queries. The changeset adds an exact-term hash/collation-key routing path for large STANDARD Boolean queries, which matches the title. The pipe-separated suffix lists release branch targets (pd, tikv, tidb, ticdc=release-8.5). That metadata adds noise, but the core description stays accurate and specific.

Full details: Description check

Explanation

The description follows the repository template. It explains the problem, implementation, tests, performance evidence, side effects, documentation impact, and release note. It also identifies pending CI and testing limitations. The issue reference uses "ref #10233" instead of the template's "close #xxx" format, but the description is otherwise complete.

Full details: Docstring Coverage

Explanation

Docstring coverage is 13.64% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 22 functions across 3 files. (1 skipped: 1 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches

Generate unit tests (beta)

• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Tip

🎯 Customize your risk assessments

Guide how CodeRabbit assesses merge risk and security architecture risk. Highlight sensitive changes and explain your security boundaries using Merge Risk Instructions and Security Risk Instructions in your review settings. Security architecture risk assessments require the Advanced plan or above.

✅ Automatic approvals, with your rules

Let CodeRabbit approve pull requests once required threads are resolved, pre-merge checks pass, and your risk limits are met. Enable Automatic approvals in review settings, with overrides for teams or file paths.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn

_{A rabbit checks each token’s key,
Then finds its matching clause with glee.
When routes do not fit, the old path stays,
Tests compare both through many ways.
The failpoint image gets its own name,
And bouncy paws applaud the change.}

CodeRabbit Verify

• Run CodeRabbit Verify to test the changes in this pull request.

Recent review info

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Repository UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: ee4ac94d-3a8b-47c4-8e06-0ca02e3fe373

Commits

Reviewing files that changed from the base of the PR and between 063a092 and 71d093c.

Files selected for processing: 1

• tests/docker/cluster_tidb_fail_point.yaml

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 2 remain after this review.

[solotzg]

/retest

[solotzg]

/retest

[solotzg]

/retest

[windtalker]

lgtm

[solotzg]

/hold

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: windtalker, xzhangxian1008

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [windtalker]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2026-11-02 09:00:16.010340886 +0000 UTC m=+484340.635491141: ☑️ agreed by windtalker.
• 2026-11-03 02:33:45.670099702 +0000 UTC m=+547550.295249971: ☑️ agreed by xzhangxian1008.

[solotzg]

/unhold

[solotzg]

/retest