[MaxTheHaters]

Summary

Scope discussion: #137. This PR remains a draft pending maintainer agreement on the direction.

FTS5 treats an unspaced Han filename as one token, so 輔導 can miss 學生情緒輔導紀錄.pdf. This adds literal basename lookup for queries containing at least two consecutive Han characters, using the existing name rows without reindexing.

query → existing FTS + meaning search
      → Han-run name prefilter → verify all query components in the basename
      → complete name/stem → partial name → existing content/meaning results

NFKC normalization, mixed case/numbers/extensions, file-kind filters and live rename/delete handling are covered. Literal filename hits point to the name rather than a semantic page. Latin-only queries and code identifier lookup keep their existing rules. This PR contains no interface-localization changes and is independent of #34.

I saw the request for maintainer agreement before implementation in CONTRIBUTING. This draft presents a reproducible fix and evaluation evidence for discussion; it is not a request to bypass that review.

Evidence

Before: name-only evaluation found the expected file first for 4 of 12 queries.
After: the same index finds it first for all 12; no embedding/model fallback is needed.

Same-index evaluation	Queries	Before R@1 / R@5 / R@10	After R@1 / R@5 / R@10
Han filenames	12	.33 / .33 / .33	1.00 / 1.00 / 1.00
Upstream synthetic	37	.92 / .97 / .97	.92 / .97 / .97
Upstream multilingual	48	.92 / .98 / .98	.92 / .98 / .98
Upstream code	41	1.00 / 1.00 / 1.00	1.00 / 1.00 / 1.00

• All 126 upstream queries produced byte-identical before/after evaluation output on generated synthetic fixtures. Personal-file sampling was omitted.
• swift test: 101 passed (95 DigUpKit, 6 LlamaRuntime, including actual model reference vectors). Nine added tests cover internal/two-character lookup, typing/settled results, exact/semantic ranking, duplicate prevention, all terms, kind filters, rename/delete, compatibility/case normalization, code isolation, and an exact name after 1,100 partial names.
• Apple M2 Pro, 16 GB; 20,000 synthetic filename rows, 2,000 matches, 20 alternating measured CLI runs after two warmups: 輔導 median 15.18 → 59.02 ms (p95 19.48 → 63.12); 情緒輔導 14.70 → 57.25 ms; Latin annual 39.91 → 39.24 ms. Times include process startup and opening the same index; the extra work returns filename hits that were previously absent.
• Reproduction commands, fixture generator and evaluation JSON are included in docs/han-filename-search.md, scripts/testbed/make_han_filenames.py and evals/han-filenames.json.

Independent regression update (2026-10-20)

• Add a regression asserting exact Traditional Chinese filenames precede partial filenames and content-only hits, with one result per file, both while typing and after submission, without vectors.
• The fixture uses the public segment insertion API and an existing standalone FTS keyword. It does not depend on the Han body tokenization or schema migration proposed by shimon-ai-dev in Fix Han full-text matching and preserve tokenizer-budgeted text tails #111; their implementation and body-search tests are not copied here.
• This update changes tests and documentation only. Interface localization remains in Add English and Traditional Chinese interface localization #34, and production search code is unchanged.
• Current focused validation: swift test --jobs 1 --filter HanFilenameTests with an isolated Xcode-build-system scratch directory: 10 tests passed. Earlier full-suite/evaluation/performance figures above are historical v0.6.1 evidence, not rerun measurements for this test-only update.

Merge Danger

Door: two-way

No schema, vector version, model, or file-content changes. Existing indexes remain readable by v0.6.1.

Blast Radius: search

Han queries with literal basename matches deliberately rank those above content-only matches. Extra filename verification costs about 44 ms in the measured 20,000-name case. Chinese full-text tokenization, Traditional/Simplified conversion and one-character substring queries are outside this change.

[ARahim3]

Thanks @MaxTheHaters, the problem is real (I said as much in #4) and your eval shows it well. I'm going to fix it in the tokenizer rather than with a name scan: FTS5's trigram tokenizer, or an ICU one, so Chinese, Japanese and Thai match inside names and inside documents alike, measured with a Chinese eval. That's a one-time schema change, which is why I'd rather not add a second path first. Closing the PR; your fixture generator and queries will be useful when I get there.