Repository navigation
Release rouge-score-rs 0.1.1 - #2
Merged
Merged
Conversation
score_batch shares Rayon's global thread pool, whose threads do not survive fork(). A child forked after the parent had used score_batch waited forever, which breaks multiprocessing and datasets.map(num_proc=...) on Linux. score_batch now remembers which process owns the pool and scores serially in forked children. Also from a post-release review: pickling keeps subclass __slots__ (the native scorer now pickles itself), the cached native scorer follows changes to the tokenizer's use_stemmer, instance-level score overrides are respected, tokenize.tokenize accepts bytes from stemmers, docs no longer overstate when score_batch is parallel, and releases now run the full test suite on the tagged commit before publishing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes found in a post-release review of 0.1.0.
score_batchhang afterfork(): Rayon's global pool threads do not survivefork(), so a child forked after the parent usedscore_batchblocked forever (affectsmultiprocessinganddatasets.map(num_proc=...)on Linux). The binding now records which process owns the pool; forked children score serially. Covered by a test with a timeout.copy.deepcopykeep subclass__slots__: the native scorer implements__reduce__, so the Python class no longer needs a custom__getstate__.use_stemmeron the scorer'sDefaultTokenizerafter construction now takes effect.score_multiandscore_batchrespect ascoremethod assigned on the instance.tokenize.tokenizedecodesbytesreturned by a custom stemmer, asrouge-scoredoes.score_batchruns in parallel, and whatAggregateScore.midis.punkt_tabdownload fails the job instead of skipping thesplit_summariesparity test; the release workflow runs the full CI on the tagged commit before publishing to PyPI; the benchmark downloads the NLTK data it needs.The Rust crate is unchanged, so only the Python package version moves to 0.1.1.