OPENNLP-1988: Read the vocabulary of a WordPiece tokenizer.json in opennlp-dl

[krickert]

Accepts a Hugging Face tokenizer.json as the vocabulary file of the deep learning components. Part 3b of the regex removal series.

• A tokenizer.json is recognized by its model.type. Only WordPiece is accepted; BPE, Unigram and other types are rejected by name, as is a continuing_subword_prefix other than ##.
• model.vocab supplies the tokens. An entry of added_tokens missing from model.vocab is added with its id. A token or id conflict is rejected naming both, and a malformed entry is rejected at added_tokens[i].
• normalizer.lowercase is read, and the constructors reject a file with a setting that disagrees with lowerCase, naming the file and both values. The other tokenizer settings are not read.
• A flat vocab.json with a value that is not an id, or a config.json handed over as a vocabulary, fails with a message naming both accepted layouts.

Tests use a hand-written tokenizer.json fixture with encodings computed by hand, rejected Unigram, BPE, prefix and conflict files, and a byte order mark before a tokenizer.json read from disk. No Hugging Face material is copied. The vocabulary rules are documented once in the document categorizer chapter; the name finder chapter and the embedder section point to it.

OPENNLP-1988

[krickert]

unstacked after the #1277 merge, so this shows only the tokenizer.json change.

A tokenizer.json is recognized by model.type and only WordPiece is accepted; added_tokens are read; normalizer.lowercase must agree with lowerCase or the file is rejected; the other tokenizer settings are not read.

Eval build: https://ci-builds.apache.org/job/OpenNLP/job/eval-tests-configurable/89/

[krickert]

Eval build was successful

[krickert]

This was a ticket for helping follow-ups stacked tix with remaining regex removal. It can hold off.

[krickert]

Thanks for the review -

Dropped the message constants and the static on the vocabulary readers. They are package-private now, since the constructor calls them and their return type is package-private. Exceptions are plain strings and JsonScan errors are no longer rewrapped; JsonScan names the offset of a bad value instead. The other static methods predate this PR, I can do those in a follow-up.

I'm applying the same logic to the other open PRs.

[rzo1]

Thx, the review comments are addressed. Please rebase onto main before merging (small conflict in JsonScan).

[krickert]

Rebased and g2g

[rzo1]

@krickert Could you rebase this onto main? Main renamed ArgumentChecks to ParamChecks (#1409), and the only conflict is in JsonScan: keep your side of both hunks and switch the ArgumentChecks calls to ParamChecks. With that, opennlp-dl builds and JsonScanTest, LoadVocabTest and SentenceVectorsDLEmbedderTest pass locally.

I would have pushed the rebase myself, but GitHub rejects pushes from maintainers to the ai-pipestream fork (same problem as before). "Allow edits by maintainers" doesn't work for organization-owned forks. Could you open future PRs from a branch on apache/opennlp or from a personal fork instead? Then we can rebase them for you.

[krickert]

Sorry I thought I transferred all the branches from my fork - I'll rebase now