Repository navigation
OPENNLP-1928: Regex removal (1/10): Replace trivial patterns with character scans - #1311
Merged
Merged
Conversation
Replaces the patterns in the POS tagger and lemmatizer context generators, BioCodec, BrownCluster and TokenPatternFeatureGenerator with character scans and adds the shared ASCII and line terminator helpers to StringUtil. The output of every site is unchanged. The type extraction of NameFinderME moves to BioCodec. Split out of #1275; the changes that alter output follow in separate parts.
rzo1
force-pushed
the
OPENNLP-1928-regex-removal-scans
branch
from
September 18, 2026 11:17
1d1b66a to
33581d0
Compare
krickert
approved these changes
Sep 18, 2026
krickert
left a comment
krickert
left a comment
Contributor
There was a problem hiding this comment.
The regex changes look good. I tested the new scans against the old regexes using the 2 million strings and they're identical.
AI found this: drop BrownCluster.splitTabs. PR #1302 introduced StringUtil.split(line, '\t'), and the tests confirm they behave identically. We can just use StringUtil and remove the private helper and its imports.
A few minor observations (no action needed): isAsciiUpperCase is private while the other ASCII checks are public, and a few methods don't have callers yet—though I see they're needed for later. The Javadoc is also spot on.
Approving now assuming we do the split tab change before merge
Contributor
Author
|
Thanks for running the 2M-string comparison! |
rzo1
marked this pull request as ready for review
September 18, 2026 11:40
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Sep 18, 2026
The AD corpus readers no longer use regular expressions. This includes the reader changes split out of OPENNLP-1928 (apache#1275): ADNameSampleStream, ADPOSSampleStream, ADSentenceSampleStream and ADSentenceStream replace their tag, whitespace, underline, hyphen, alphanumeric, metadata and punctuation patterns with character scans over the StringUtil helpers. ADSentenceStream.SentenceParser reads node, leaf and bizarre-leaf lines and the fallback check with cursor scans, ADSentenceStream recognizes the sentence, title, box, paragraph and text tags by name, and the package-private ADMetadata reads the text id, the paragraph id and the source for ADNameSampleStream and ADSentenceSampleStream. The scans follow the format, not the accidents of the old patterns: an AD tree line without a tag keeps its full lexeme, the first lexeme character is no longer lost, paren-in-group node tails and quoted-bracket tags are read, and whitespace follows the Unicode definition. Hyphenated words are preserved by default; the ADTokenSampleStreamFactory option and the CLI chapter describe the switch. A leaf without a functional tag in ADChunkSampleStream is tagged with its lexeme, as the POS reader does. The corpora chapter states the reader contract. On the real corpora the output is unchanged: all AD streams over Bosque_CF_8.0 and FlorestaVirgem_CF_3.0 are byte-identical to the previous readers, and the ArvoresDeitadasEval training data hashes are the same. Squashed from the review rounds of apache#1276 for the rebase onto apache#1311.
rzo1
commented
Sep 18, 2026
…ad of a message constant
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Sep 18, 2026
The AD corpus readers no longer use regular expressions. This includes the reader changes split out of OPENNLP-1928 (apache#1275): ADNameSampleStream, ADPOSSampleStream, ADSentenceSampleStream and ADSentenceStream replace their tag, whitespace, underline, hyphen, alphanumeric, metadata and punctuation patterns with character scans over the StringUtil helpers. ADSentenceStream.SentenceParser reads node, leaf and bizarre-leaf lines and the fallback check with cursor scans, ADSentenceStream recognizes the sentence, title, box, paragraph and text tags by name, and the package-private ADMetadata reads the text id, the paragraph id and the source for ADNameSampleStream and ADSentenceSampleStream. The scans follow the format, not the accidents of the old patterns: an AD tree line without a tag keeps its full lexeme, the first lexeme character is no longer lost, paren-in-group node tails and quoted-bracket tags are read, and whitespace follows the Unicode definition. Hyphenated words are preserved by default; the ADTokenSampleStreamFactory option and the CLI chapter describe the switch. A leaf without a functional tag in ADChunkSampleStream is tagged with its lexeme, as the POS reader does. The corpora chapter states the reader contract. On the real corpora the output is unchanged: all AD streams over Bosque_CF_8.0 and FlorestaVirgem_CF_3.0 are byte-identical to the previous readers, and the ArvoresDeitadasEval training data hashes are the same. Squashed from the review rounds of apache#1276 for the rebase onto apache#1311.
jzonthemtn
approved these changes
Sep 18, 2026
Contributor
|
+1 Thanks @rzo1 |
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Sep 18, 2026
The AD corpus readers no longer use regular expressions. This includes the reader changes split out of OPENNLP-1928 (apache#1275): ADNameSampleStream, ADPOSSampleStream, ADSentenceSampleStream and ADSentenceStream replace their tag, whitespace, underline, hyphen, alphanumeric, metadata and punctuation patterns with character scans over the StringUtil helpers. ADSentenceStream.SentenceParser reads node, leaf and bizarre-leaf lines and the fallback check with cursor scans, ADSentenceStream recognizes the sentence, title, box, paragraph and text tags by name, and the package-private ADMetadata reads the text id, the paragraph id and the source for ADNameSampleStream and ADSentenceSampleStream. The scans follow the format, not the accidents of the old patterns: an AD tree line without a tag keeps its full lexeme, the first lexeme character is no longer lost, paren-in-group node tails and quoted-bracket tags are read, and whitespace follows the Unicode definition. Hyphenated words are preserved by default; the ADTokenSampleStreamFactory option and the CLI chapter describe the switch. A leaf without a functional tag in ADChunkSampleStream is tagged with its lexeme, as the POS reader does. The corpora chapter states the reader contract. On the real corpora the output is unchanged: all AD streams over Bosque_CF_8.0 and FlorestaVirgem_CF_3.0 are byte-identical to the previous readers, and the ArvoresDeitadasEval training data hashes are the same. Squashed from the review rounds of apache#1276 for the rebase onto apache#1311.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 1 of the regex removal (OPENNLP-1926). Split out of #1275, which mixed output-preserving rewrites with behavior changes; this PR holds only the rewrites whose output is unchanged.
What changes
DefaultPOSContextGenerator,DefaultLemmatizerContextGenerator: the[A-Z]and[0-9]capital and digit features are ASCII range checks (StringUtil.containsAsciiUpperCase,containsAsciiDigit). They stay ASCII-only because trained models depend on them.TokenPatternFeatureGenerator: the[^a-zA-Z]check is a character scan.BioCodec: the typed outcome pattern(.+)-\w+islastIndexOf('-')plus a word character scan.extractNameTypemoves fromNameFinderMEtoBioCodec; both sequence validators call it there. It was package-private.BrownCluster: the tab pattern split isStringUtil.split(line, '\t')from OPENNLP-1955: Replace literal String.split calls with a regex-free StringUtil.split #1302, same semantics.StringUtil
New helpers:
containsAsciiUpperCase,containsAsciiDigit,isAsciiLetter,isAsciiLowerCase,isAsciiDigit,endOfAsciiDigits,isLineTerminator,indexOfLineTerminator. Some are only used by later parts (AD readers in #1276, CoNLL-U in #1314, the glob matcher in #1278).Sequence
Commits by @krickert from #1275, regrouped.