Skip to content

perf: route ASCII paragraph separators without property lookups - #18

Merged
xclud merged 2 commits into
olutter:mainfrom
palmoni5:codex/upstream-bidi-separator-routing
Nov 14, 2026
Merged

xclud merged 2 commits into
olutter:mainfrom
palmoni5:codex/upstream-bidi-separator-routing

Conversation

@palmoni5

Copy link
Copy Markdown
Contributor

Paragraph scanning needs only to identify paragraph separators, but currently looks up the full bidi character type for every ASCII code unit. Recognize the five ASCII paragraph separators directly (LF, CR, U+001C..U+001E), while preserving the full Unicode table lookup for everything else.

Use the same private predicate in both BidiString.fromLogical and the deprecated splitStringToParagraphs scanner. Preserve the original codeUnits iteration, paragraph grouping, rendering behavior, and source-index APIs. CR and LF remain separate paragraph boundaries. This change is independent of the supplementary-Unicode and source-mapping proposals and is based directly on upstream main ebef4fb.

Validation:

  • flutter analyze: no issues.
  • dart test: all 5 tests pass on this independent branch.
  • Test every ASCII character against the original Unicode classification in both public paragraph scanners; cover repeated boundaries, CRLF, non-ASCII boundaries, and the UTF-16 offset after an emoji.
  • Independent differential comparison against upstream main: identical visual output, paragraph metadata, and existing index/output-list behavior for 18,897 inputs. The corpus includes all 16,384 ASCII pairs, 512 mixed-script boundary cases, empty input, and 2,000 seeded random 100-scalar strings with Hebrew, combining marks, directional controls, Unicode separators, and supplementary characters.
  • Add a benchmark covering long ASCII, mixed Hebrew/Latin, and many ASCII paragraph boundaries. CI runs full analysis, tests, and the benchmark.

Performance context: paired measurements on the integration fork (which already bypasses other ASCII normalization costs) showed ASCII 259-268 to 189-193 us/op, mixed text 746-748 to 710-719 us/op, and separator-heavy text 122-126 to 99-102 us/op. On the original upstream implementation, normalization dominates and end-to-end measurements were mixed; no general whole-paragraph speedup is claimed for that baseline.

@xclud
xclud merged commit bd4c819 into olutter:main Nov 14, 2026
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants