[proger]

I benchmarked three branches (main, experiments and #4) with this patch on a file with 100k lines.

I got the best results with code from #4 using AccurateSrxTextIterator clocked at 141.82s. tokenize_text from nlp_uk clocks at 54.58s. If I disable word segmentation (remove -u -w) nlp_uk clocks at 35.324s.

This code generates a bit less sentences on that dataset: 84452 from nlp_uk vs 84018 in choppa. See sentence-wise diff.

Here's the full log: https://gist.github.com/proger/2fa3ead52dc78b7d582cd356d9f423e9

[dchaplinsky]

Superseded by scripts/benchmark.py on main (which also byte-diffs against the Java segment output), and the CLI improvements from this PR (line-by-line mode, iterator selection) landed as part of the choppa console script in #6 — thank you!

Fresh numbers on 100k lines / 11 MB: 4.7s for SrxTextIterator (vs 141.8s for the best 2022 configuration and ~55s for nlp_uk), output byte-identical to Java. The sentence-count gap this benchmark exposed (84,018 vs 84,452) was the thread that led to the 100-char window bug — this PR earned its keep.