Repository navigation
Finish the port: Java-identical matching, current LT rules, 24-language test suite, 1.0.0 packaging - #6
Merged
Conversation
The old JavaMatcher searched only a max_lookaround_len (100 char) window from the region start, silently dropping any break further away — the source of the 2022 'real-world discrepancies' vs Java and the sentence count gap in the nlp_uk benchmark. The new RuleMatcher uses pattern.search(text, pos)/pattern.match(text, pos) directly, replicating java.util.regex.Matcher semantics: - find() restarts at the end of the previous match, +1 only after zero-width matches (Java's implicit empty-match bump); - the exception check pattern.match(text, break_position) is exactly Java's useTransparentBounds(true) + region + lookingAt(), since Python keeps the left bound transparent for lookbehind; - rule patterns are now compiled WITHOUT MULTILINE, matching Java, and Java-only constructs ((?U), \h, \v) are translated with escape-aware rewriting; SrxDocument(pattern_flags=...) is the analog of segment 2.0.4 defaultPatternFlags for opting into extra flags such as regex.M. Output is byte-identical to segment-2.0.3 (ultimate algorithm, current LanguageTool segment.srx) on a 100k-line/11MB Ukrainian corpus: 43,569 segments, 5.4s vs 29.6s for Java. Based on the SimpleRuleMatcher approach proposed by Jarek Lipski in #4, with the find() advancement fixed to Java semantics. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Refresh from the 2022-08-03 snapshot to commit d24cd528 — 142 upstream commits, including 13 Ukrainian rule updates by Andriy Rysin, the JDK>=19 (?U) flag fixes (now handled by the regex translator), and the long-standing [Тт]]ел typo fix. Add PROVENANCE.md with refresh instructions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add SrxTextIteratorReaderTest (whole abstract suite with buffer=60, margin=10, per upstream) and testMaxLookbehindConstructLength. The streaming suite immediately exposed a port bug: Java resets start to 0 after textManager.readText(start) shifts the buffer, but the port kept the stale start_position, truncating output after the first buffer re-read. Also fix a self.result typo in TextManager.read. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ses) scripts/extract_lt_tests.py converts LT's Java *SentenceTokenizerTest files into JSON fixtures (tests/data/lt/), resolving each testSplit call's paragraph mode (_one/_two) including mid-method setSingleLineBreaksMarksParagraph flips, Java string literal concatenation/escapes, and skipping non-SRX tokenizers (Chinese). All 848 extracted cases pass against the bundled 2026-05-16 segment.srx. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- pyproject.toml (PEP 621/639), distribution name choppa-srx (the bare name is taken on PyPI), import name stays choppa; Python 3.9-3.14 - proper argparse CLI as a 'choppa' console script (line-by-line mode and iterator selection based on Volodymyr Kyrylov's PR #5) - top-level exports (SrxDocument, SrxTextIterator, AccurateSrxTextIterator), py.typed marker, version 1.0.0 - GitHub Actions: test matrix + tag-triggered PyPI trusted publishing - README rewrite: compatibility statement, performance numbers, rules provenance; CHANGELOG added Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ence - ITERATORS registry and bundled-data paths now live in the package (choppa.iterators.ITERATORS, choppa.DEFAULT_SRX_RULESET, choppa.SRX_2_XSD); CLI, benchmark and tests import them instead of re-declaring - translate_java_regex moved to choppa/utils.py (the regex-mangling module) and rewritten as an escaped-pair alternation — no variable lookbehind, no 100-pair cap, and escaped constructs (\\h, \(?U\)) are now handled correctly - regex_cache key includes pattern_flags; language-rule lookups are memoized per (language_code, cascade) — removes ~9% overhead in the CLI line-by-line mode without touching the algorithm - CLI default mode streams the input through SrxTextIterator's buffer instead of slurping the whole file; --buffer-length exposed - extract_lt_tests.py: single skip_string_literal scanner replaces four copies, shared mode-call helper, token-boundary definition detection (adds 'protected'), and the SRXSentenceTokenizerTest special case now flows through the general skip-reporting; fixtures regenerate byte-identically - reader test overrides TEXT_LONGER_THAN_BUFFER_RESULT instead of re-implementing perform_test; LT test file uses lru_cache and drops cosmetic __name__/__doc__ fiddling; dead 'document' slot removed from RuleMatcher; benchmark argparse description was an empty string Verified: 112 tests + 848 LT subtests green; output still byte-identical to Java segment on the 11MB corpus; CLI stream mode output equals whole-string mode. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- tests/test_utils.py: 6 new tests for translate_java_regex (whitespace classes, (?U) dropping, escaped-construct handling, exact Java \h/\v code point coverage incl. U+180E) - README gains a Performance and verification section: byte-identity vs segment-2.0.3 confirmed on 7 corpora / ~136k segments (uk news from Militarny, uanews.dp.ua, Liga.net; Pride and Prejudice, Moby-Dick, Sherlock Holmes for en) with timings and reproduction instructions (incl. the CRLF pitfall) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- docs/api.md: public API surface and stability promise, constructor parameters, streaming semantics and the buffer constraint, language key conventions, CLI reference, error behavior (unknown-language fallback verified empirically) - docs/design.md: the byte-compatibility contract and how it is met — regex-package choice, Java->Python Matcher semantics mapping table, rule translator, ^/MULTILINE decision, finitization, algorithm choice, performance model, three-layer testing philosophy - docs/history.md: SRX / segment / LanguageTool lineage, the 2022 port and issue #22 story, the 2026 completion - README links the docs; CONTRIBUTING updated (py3.9, editable install, byte-compatibility requirement for behavior changes); docs shipped in the sdist Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Segments preserve input bytes exactly, so echo's trailing \n stayed inside the last segment and printed as an extra blank line, tripping the wc -l assertion. Use printf and diff against the exact expected output, which also asserts the segmentation itself. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This wraps up the port that started in 2022 (see loomchild/segment#22).
The root cause of the old discrepancies
JavaMatchersearched only amax_lookaround_len(100-char) window from the region start, so any break located further than 100 characters away was silently dropped and the rest of the text came back as a single segment. This affectedSrxTextIteratoron every branch (including #3 and #4) and explains both the "discrepancies on real-world texts" from 2022 and the sentence-count gap in the #5 benchmark.What changed
RuleMatcheris now pure positional matching (pattern.search(text, pos)/pattern.match(text, pos)), following theSimpleRuleMatcherapproach @loomchild proposed in Implement simple rule matcher #4, withfind()advancement matchingjava.util.regex.Matcherexactly: restart at the previous match end, +1 only after zero-width matches (the unconditional +1 in Implement simple rule matcher #4 could skip adjacent matches).JavaMatcheris gone.exception_pattern.match(text, break_position)— Python's positional matching keeps the left bound transparent for lookbehind, which is precisely Java'suseTransparentBounds(true)+lookingAt().MULTILINE(Java semantics for^);SrxDocument(pattern_flags=regex.M)restores the old line-start behavior (analog of segment 2.0.4defaultPatternFlags).\h→\p{H},\v→\p{V}, and the(?U)inline flag (JDK≥19 fixes in current LT rules) is dropped since Python is Unicode-aware by default.start = 0reset after the buffer shift, truncating output after the first re-read. The full upstream streaming suite (buffer=60, margin=10) is now ported and passing.segment.srxupdated from the 2022-08-03 snapshot to LT master 2026-05-16 (142 commits, 13 uk rule updates), withPROVENANCE.md.scripts/extract_lt_tests.pyregenerates the fixtures) — all passing.pyproject.toml, distribution namechoppa-srx(barechoppais taken on PyPI; import name unchanged),choppaconsole script (line-by-line mode from Benchmarking against nlp_uk #5 by @proger), py.typed, CI matrix 3.9–3.14, tag-triggered trusted publishing.Verification
Byte-level comparison against
segment-2.0.3(ultimatealgorithm, current LTsegment.srx, JDK 17 and 24 produce identical output) on real Ukrainian news text:(vs. 141.8 s for the best 2022 configuration in #5, and ~55 s for nlp_uk.)
Full suite: 112 tests + 848 LT subtests, green on 3.9–3.14.
🤖 Generated with Claude Code