Skip to content

perf: use dense ASCII and direct Hebrew bidi classification - #19

Merged
xclud merged 2 commits into
olutter:mainfrom
palmoni5:codex/upstream-character-classes
Nov 14, 2026
Merged

xclud merged 2 commits into
olutter:mainfrom
palmoni5:codex/upstream-character-classes

Conversation

@palmoni5

Copy link
Copy Markdown
Contributor

getCharacterType probes the large characterTypes map for every character, including ASCII punctuation, numbers and Latin text, and the contiguous Hebrew alphabet. These dominate many mixed Hebrew/Latin documents.

Use an immutable 128-entry array containing the existing ASCII classes and return R directly for U+05D0..U+05EA. Preserve the public characterTypes map and its fallback for every other input, including supplementary and invalid codepoints. This branch is independent of the supplementary Unicode, normalization, source mapping and paired-bracket changes; it targets upstream main at ebef4fb without prerequisites or an SDK/API change.

Add exhaustive BMP comparisons, boundary/fallback checks, a mixed-text regression, a reproducible benchmark and full analysis/test CI.

Validation:

  • flutter analyze: no issues.
  • dart test --reporter expanded: all 5 tests passed, including all 65,536 BMP inputs.
  • Independent differential harness against unchanged upstream: all 1,114,112 codepoint inputs and four signed/out-of-range sentinels matched, exercising all 23 enum classes. All 20,384 ASCII-pair and seeded Hebrew/mixed cases retained visual output, indices and the existing logicalToVisual2 output-list behavior.
  • Two sequential baseline/candidate benchmark pairs on the same Windows host, in microseconds per batch:
Case Baseline 1 Candidate 1 Baseline 2 Candidate 2
5,400 Hebrew classifications 113.1 52.8 80.1 60.1
6,400 ASCII classifications 98.1 67.5 95.7 69.0
4,800 mixed classifications 76.1 49.9 74.0 52.4
Complete mixed paragraph 827.8 770.7 869.8 873.1

Classification batches improved in both runs; the complete-paragraph timings are noisy and do not establish a consistent end-to-end improvement. Benchmark checksums matched. No machine-dependent timing threshold is imposed.

@xclud
xclud merged commit cc0962c into olutter:main Nov 14, 2026
1 check failed
@palmoni5
palmoni5 deleted the codex/upstream-character-classes branch November 14, 2026 16:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants