Repository navigation
OPENNLP-1974: Normalize complete Unicode emoji sequences - #1313
Conversation
ea2d4d1 to
9dc83d4
Compare
9dc83d4 to
dab9ea7
Compare
|
Unicode emoji file was missed on this, it's back now so it's working. |
krickert
changed the title
Use a code-point matcher and pinned Unicode Emoji 18 data. Preserve non-emoji text and malformed candidates. Document training and inference implications. Red: 6 of 8 intended-behavior checks failed on the former implementation; 11 Emoji 18 cases failed with the Emoji 17 inventory. Green: 50 focused tests pass, including all 3963 bundled sequences.
Append the Unicode V3 license and Emoji 18 source attribution to the runtime artifact metadata. Include the data license when the runtime JAR is distributed separately.
dab9ea7 to
5ae9fbe
Compare
krickert
left a comment
krickert
left a comment
There was a problem hiding this comment.
From reviewing it, caught that Emoji v18 just came out last week. Includes the thumb innovations from the latest spec.
…vadoc Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
@krickert, two questions about the bundled emoji data before we merge:
|
I'll do a total audit to see what the dependency is. I remember seeing most of them were thumbs of multiple colors, but I'll double check
I think so, but totally not important (I don't think this is a heavily used feature, but will become an incredibly helpful one in the future). But any pushback on emjois I won't battle :) No problem - 17 is coming back. Made it a draft for a quick sec to make sure the other nits are still in though... I thought of it as a quick property update, but even that should be it's own PR. Noted. |
|
@krickert I would treat a version update 17 -> 18 similar to a "code" dependency version update, so it is trackable in the release notes and doesn't hide in a normalizer ticket :) |
|
@krickert Any updates here? |
|
\I had 2 left from last night before I stopped, this was one of 'em. This is gettng the split today, and a v18 branch will be created w/o a PR yet (assumed we'll do v18 for 3.1+). I should be able to get to it tonight / tomorrow AM |
Use Unicode Emoji 17.0 data for emoji removal: 3,944 sequences and 10 component ranges. Pin the release, source checksums and sequence count in the generator. Load the data lazily and reject malformed entries. Keep emoji next to stray joiners, modifiers, selectors, flag letters or tags, and remove other complete emoji in the same run. Remove one extra U+FE0F after a complete emoji with that emoji. Skip non-emoji text before the trie lookup. Preserve the old output under opennlp.compat.mode=LEGACY with a cursor scan instead of regex. Compare it with the old pattern over 5,000 randomized inputs, and test the normalizer chain in both compatibility modes. Remove the class deprecation and use it in the default normalizer chain. Document the behavior in Javadoc and the language detector chapter, and update the runtime jar LICENSE and NOTICE for the bundled Unicode data. Assisted-by: Claude Code 2.1.281 Co-authored-by: Kristian Rickert <kristian@apache.org> Co-authored-by: Claude <noreply@anthropic.com> Reviewed-by: Kristian Rickert <kristian@apache.org> Signed-off-by: Kristian Rickert <kristian@apache.org>
|
Pushed the fixes and moved the data back to Emoji 17.0. The Emoji 18.0 update is tracked in #1347 for 3.1. LEGACY mode keeps the old behavior without using regex. The tests compare its output against the old pattern. Removed the deprecation and added the normalizer to the default chain. Emoji next to stray components are preserved; other complete emoji are removed, including one extra U+FE0F after them. Updated the Javadoc and Unicode license notices, and moved the tests to opennlp-runtime. Eval: https://ci-builds.apache.org/job/OpenNLP/job/eval-tests-configurable/91/console |
- Trie nodes keep sorted primitive arrays instead of HashMap<Integer, Node>, so a lookup does not box. - Lazy loading uses a holder class instead of double-checked locking. - The normalizer checks the candidate start set before decoding a second time. - Javadoc names the pre-3.0.0 regex once instead of "the former pattern". - HexCodePoints no longer claims to be shared by the other data loaders. - The generator reads each source file once and accepts a bare output name. - The characterization tests are parameterized so a failing input is named. - The docbook entry keeps two sentences in the table and moves the LEGACY note below it.
0b4f549 to
69450f9
Compare
… HexCodePoints Confusables and EmojiAnnotations used their own Integer.parseInt(..., 16) loops. They now use HexCodePoints like UnicodeEmojiSequences, so an out of range code point fails with the line number instead of a bare exception from appendCodePoint.
emoji-test.txt lists 1,029 minimally-qualified sequences, mostly gendered and skin tone ZWJ sequences written without U+FE0F after a later element, such as U+1F64B U+1F3FF U+200D U+2640. They were kept whole as malformed candidates. The generator now keeps them next to the fully-qualified ones, so the inventory holds 4,973 sequences. Unqualified sequences, whose first code point lacks a needed U+FE0F, are still kept as text.
- UnicodeEmojiSequences: adding a sequence is an instance method of the trie node instead of a static helper taking the root. - UnicodeEmojiSequenceGenerator: main creates an instance; the helpers are instance methods. - Tests: helpers called only from test methods are instance methods. Factories (getInstance, load, parse), the HexCodePoints utility, JUnit method sources and the helpers they call stay static.
|
Note for the 3.0.0 release notes: since Until then, users of the existing model can get the old features back with Also worth noting: |
The normalizer data loaders read code point sequences separated by any whitespace run, with leading and trailing whitespace ignored, instead of single spaces only.
HexCodePoints tokenizes a sequence with StringUtil.splitOnUnicodeWhitespace, so the confusables and annotation loaders and the emoji trie accept tabs and repeated spaces between code points and ignore surrounding whitespace. The single space separator constant is gone. The getInstance Javadoc of UnicodeEmojiSequences names the error the holder idiom produces.
|
I did a minor change - only single whitespace (' ') was checked, changed it to apply to all unicode whitespace. Sourceforge was down, which is why tests were red. |
2 participants
Replaces the character-range deletion in
EmojiCharSequenceNormalizerwith a code-point scanner that removes complete Unicode emoji sequences and leaves letters, CJK and text symbols in place.EmojiSequences.txt, generated bydev/UnicodeEmojiSequenceGeneratorfrom the release files with pinned SHA-256 checksums and count. A minimally-qualified sequence lacks U+FE0F after a later element only and is removed like a fully-qualified one; an unqualified sequence, with no U+FE0F after the first symbol, is left as text. The Emoji 18.0 update is Update the bundled Unicode Emoji data to 18.0 #1347, for 3.1.opennlp.compat.mode=LEGACYthe normalizer reproduces the old output with a cursor scan: every hyphen, U+E000 to U+FFFF and nearly every supplementary character removed, BMP emoji such as U+231A kept. A characterization test compares it with the old pattern on 5,000 randomized inputs.OPENNLP-1974