Repository navigation
Unicode and character tokenizer profiles (0.2.0) - #4
Merged
Merged
Conversation
The rouge-score tokenizer keeps only [a-z0-9], so ROUGE is zero for most languages. Users work around it with Python tokenizers, which also turns off the parallel path. Two opt-in, versioned profiles now run natively: - unicode-v1: lowercase, NFKC, UAX #29 words that contain a letter or a number, for languages that separate words with spaces. - character-v1: lowercase, NFKC, one token per grapheme cluster, for character-level ROUGE such as in Chinese. The unicode-segmentation and unicode-normalization versions are pinned so a profile keeps its behaviour; both use Unicode 17.0, which needs Rust 1.85. The default tokenizer and its scores are unchanged.
str::to_lowercase and char::is_alphanumeric follow the Unicode version of the compiler, so the same profile could tokenize differently depending on the Rust release a wheel was built with. Case folding and the alphabetic/number test now come from ICU4X 2.3 (Unicode 17.0), pinned like the normalization and segmentation crates. Normalization is NFKC, full case folding, then NFKC again. Folding before NFKC left compatibility capitals such as U+1D400 uppercase, and folding is context-free, unlike lowercasing with its final-sigma rule. ICU4X needs Rust 1.88.
icu_casemap and icu_properties accept any 2.3.x data release, so a downstream lockfile could pick newer tables and change tokens.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
rouge-score's tokenizer keeps only[a-z0-9], so ROUGE is zero for Russian, Arabic, Hindi, Chinese and most other languages. This adds two opt-in, versioned tokenization profiles that run natively, including in parallelscore_batch:UnicodeTokenizer(unicode-v1): NFKC, full Unicode case folding, NFKC again, then the words found by the Unicode word boundary rules (UAX #29) that contain an alphabetic character or a number. For languages that separate words with spaces.CharacterTokenizer(character-v1): the same normalization, then one token per grapheme cluster that contains an alphabetic character or a number. For character-level ROUGE, a common convention for Chinese.The default tokenizer and its scores are unchanged.
A profile name must always mean the same tokens, so all Unicode data comes from exactly pinned crates (unicode-normalization 0.1.25, unicode-segmentation 1.13.3, ICU4X 2.3.0 and its data crates, all Unicode 17.0) rather than from the standard library, whose tables follow the compiler version. ICU4X raises the minimum Rust version to 1.88.
Tests: token fixtures for several scripts, normalization idempotence, Unicode 17 characters, and native scoring checked against
rouge-scorefed the same tokens through its custom-tokenizer path (1,000 random multilingual pairs per profile, plusscore_batch,score_multiand pickling).On 5,000 pairs of 60-word texts, compared with
rouge-scoreplus a Python tokenizer: Russian withunicode-v1is about 11× faster per call and 50× withscore_batchon 8 cores; Chinese withcharacter-v1about 50× and 200×.