OPENNLP-1946: Add BoundedUrlCharSequenceNormalizer for whole URLs and document the limits of the default

[krickert]

Based on main, independent of the OPENNLP-1928 stack.

UrlCharSequenceNormalizer is part of the default language detector chain, so its output is part of the features existing models were trained on. It stays as it is. Its Javadoc now states the three limits found with the WHATWG URL test data, and a test table records the current output on those inputs:

• A URL is removed only up to the first character outside the ASCII body set: http://f:21/ gives " :21/", http://example.com/%20foo gives " %20foo", and http://[::1]/ is untouched.
• The scheme has no left boundary: git+https://github.com/foo/bar gives "git+ ".
• The email pass matches inside the userinfo of a URL with another scheme: telnet://user:pass@foobar.com:23/ gives "telnet://user: :23/".

The email scan moved to a package-private helper, MailAddressScan, that both normalizers use; the default's output and its serialVersionUID (equal to the serialver value before and after) are unchanged.

The opt-in variant

BoundedUrlCharSequenceNormalizer, a separate final class with getInstance() and readResolve, not in the default chain. A separate class rather than a flag on the existing one, so the serialized form and the behavior of the class inside trained models do not change, and opting in is passing the other instance to the context generator. Rules, all cursor scans by code point:

• The scheme is http or https in any letter case, followed by ://, at the start of the text or after a code point that is neither a letter, a digit, nor one of _ + - . :. So git+https:// and blob:https:// are not URLs of their own, and neither is see:http://.
• The body runs to the end of the text or the first Unicode whitespace, control character, unpaired surrogate, typographic quotation mark, or one of < > ". It includes a port, userinfo, percent escapes, a bracketed IPv6 host, and non-ASCII host names and paths, supplementary characters included.
• Trailing sentence punctuation (. , ; : ! ?) and a closing bracket without its opening bracket inside the body are given back (the bracket balance is counted once per URL, so a long run of closers costs linear time), so (http://example.com/a). gives "( )." and http://example.com/a(b) is removed in full.
• A URL with another scheme (telnet://, redis://, ftp://, ...) is left as it is, and no email address is matched inside it. Email addresses elsewhere are matched as in the default.

Out of scope, as in the ticket: host validation, the unusual forms http:foo and http:/x, and schemes without :// such as tag:.

Tests

BoundedUrlCharSequenceNormalizerTest, 105 cases: the WPT-style URLs from the ticket removed in full, other schemes untouched, left boundaries, trailing punctuation and brackets, every kind of body delimiter (space, tab, newline, no-break space, ideographic space, U+2028, U+0085, control characters, an unpaired surrogate on either side, typographic quotes), text without a URL, URLs at the start, the end, twice, and nested, email addresses next to and inside URLs, null, empty, any CharSequence, and that deserialization yields the shared instance. UrlCharSequenceNormalizerTest gained 16 rows that pin the default's output on the same inputs. Six more cases feed a URL followed by 1000 to 5000 closing brackets through a CharSequence that counts character reads, so a return to quadratic trimming fails the test.

The WPT data (url/resources/urltestdata.json) is not vendored: a copy would need a LICENSE and NOTICE entry for its 3-clause BSD license, which is left to the maintainers; the tests hold a curated subset instead.

opennlp-runtime 2680 and opennlp-tools 546 tests with checkstyle, -Dopennlp.forkCount=1.

Manual

langdetect.xml: the normalizer row states what the default does with a port or a non-ASCII host and names the variant, and a new section "Selecting URL normalization" describes the variant's rules and how to opt in through a custom LanguageDetectorFactory.

Before merge

• No eval build needed: the default chain is unchanged (pinned by tests). A model trained with the variant must be decoded with the variant.

OPENNLP-1946

[rzo1]

The default UrlCharSequenceNormalizer output is wrong (ports, userinfo, git+https), so please fix it in place instead of adding BoundedUrlCharSequenceNormalizer as a new public class nobody gets by default, and keep the old output under CompatibilityMode.LEGACY from #1312, the same way #1313 handles the emoji normalizer. That also means rebasing onto #1312, running the langdetect eval build before merge, and retitling to the user-visible change, e.g. "UrlCharSequenceNormalizer removes whole URLs including ports, userinfo and non-ASCII hosts". A trailing ASCII apostrophe is currently swallowed with the URL ('http://example.com/a' gives "' "), so please give it back like the other trailing punctuation and pin it with a test. Please reuse StringUtil.isAsciiLetter from #1311 instead of the private copy, add full Javadoc to isHttpScheme, and move ASCII_ALNUM out of MailAddressScan to a neutral place. Finally, drop the "(OPENNLP-1946) … kept as they are" history from the Javadoc and tests, and rewrite the langdetect.xml row in the style of the emoji row from #1313.

[rzo1]

Closed in favor of #1315, which applies the review here: UrlCharSequenceNormalizer is fixed in place, the earlier output stays available under CompatibilityMode.LEGACY, and BoundedUrlCharSequenceNormalizer is gone. Your commit is kept as the first commit there. I couldn't push to the fork branch, hence the new PR.

[krickert]

Danke, @rzo1