OPENNLP-1942: Test the shared deep-learning encoder and fix the token type javadoc

[krickert]

Follow-up to #1165 on OPENNLP-1942, based on main and independent of the rest.

#1165 routed DocumentCategorizerDL, NameFinderDL and SentenceVectorsDL through one AbstractDL.encodeTokens. The per-component tests that had checked the token and id arrays were removed along with the duplicated code they covered, leaving one encoding case in CreateTokenizerTest. This adds a test class for that method, plus two javadoc fixes found while writing it.

AbstractDLEncodeTokensTest checks the tokens, ids, attention mask and segment ids of AbstractDL.encodeTokens in 36 cases without a model file:

• empty input, ASCII and non-ASCII whitespace, a StringBuilder input
• known words, unknown words, wordpiece continuations, and mixed sequences
• precomposed and decomposed accents and a word-final Greek capital sigma, each under both case settings
• a CJK run split into single characters (known and unknown, cased and uncased)
• a supplementary-plane symbol as a known token, as an unknown token, and inside a word, where it makes the complete word unknown
• known punctuation, a punctuation run yielding one unknown token per character, and [CLS] as input text, which is split as text and not mapped to the special id
• a removed control character and a removed zero width space that joins two words
• the word length limit: 100 code points in 101 chars is still looked up, 101 code points is unknown
• the RoBERTa special tokens, and a RoBERTa token map that falls back to the BERT unknown token

Ids are named constants shared between the token maps and the expectations, and include 0 and Integer.MAX_VALUE so a sign or ordering slip fails. The mask and segment arrays are derived from the token count. A further test checks that repeated calls return independent arrays.

ModelFreeDL is the model-free subclass both test classes share; CreateTokenizerTest covers creation only.

Javadoc fixes:

• Tokens.mask and Tokens.types now say what the encoder writes: the attention mask is all 1, the segment ids are all 0 for a single segment. The old text said types were "typically all 1".
• WordpieceEncoder links SpecialCasing.txt where it mentions the Unicode full case mapping, matching the WordBreakProperty.txt reference already there.

No production behavior changes.

Verification, offline, with checkstyle:

./mvnw -o -pl opennlp-core/opennlp-ml/opennlp-dl test -Dopennlp.forkCount=1

AbstractDLEncodeTokensTest 38 tests, CreateTokenizerTest 11 tests, no failures.

[rzo1]

This cannot be 1885 because it is already in the proposed M6 release candidate. It needs a new Jira.

[krickert]

This cannot be 1885 because it is already in the proposed M6 release candidate. It needs a new Jira.

I'll get to this tonight

[rzo1]

Thanks for the new Jira. Two things are still open from my earlier comment:

• Both commits (c83b1f4a5, 88ca5f0c0) still say OPENNLP-1885. Please reword them to OPENNLP-1942 (or we squash on merge with the PR title).
• OPENNLP-1942 has 3.0.0-M6 as fix version. This is not in M6, please keep only 3.0.0.

Please check the commit messages and fix versions of your other open PRs that were moved to a new Jira likewise.

Missing test case: no input with a supplementary-plane character (emoji / surrogate pair) or a CJK character. Please add one for each (unknown, and CJK split).

Details inline.

[krickert]

Thanks for the review. Pushed 3d5943c, rebased on current main:

• Both earlier commits are reworded to OPENNLP-1942; the review changes are a third commit on top.
• The fix version on OPENNLP-1942 is 3.0.0 only.
• Supplementary-plane and CJK cases are in, unknown and split. While adding them I also covered a known emoji token, an emoji inside a word, punctuation isolation, special token text in the input, control and format character removal, and the 100 code point word limit. The PR description lists them.

On the other PRs: #1236 is titled OPENNLP-547 with OPENNLP-1888 commits, and #1238 has no ticket in its commit subjects. I will fix both when I next touch those branches.