Repository navigation
fix: write and read a chunk whose encoded bytes pass 2 GiB - #631
Merged
Merged
Conversation
dfa1
force-pushed
the
fix/629-fsst-large-offsets
branch
from
November 1, 2026 18:30
dd10b60 to
f2898bd
Compare
Three int limits stood between a large chunk and the u32 segment length the format allows (up to 4 GiB): - FsstEncodingEncoder kept the code offsets in an int[], overflowing once the compressed codes passed 2 GiB. They are a long[] now, as Rust's fsst_compress switches to i64 offsets past i32::MAX. - VortexWriter wrote a segment through one ByteBuffer view, which cannot pass 2 GiB. It writes the segment in 1 GiB slices, still zero-copy. - SerializedArrayDecoder narrowed the segment length to int; it stays long. Found by the Raincloud corpus: peoples-speech-clean-validation (2.3 GB audio column in one 8192-row block) now imports, and its CSV export is byte-identical to the vortex-jni-written file's. The cross-check needs ~2.5 GB of input, too big for CI runners, so it gets a new opt-in `large` tag, excluded like `raincloud`. Closes #629 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dfa1
force-pushed
the
fix/629-fsst-large-offsets
branch
from
November 1, 2026 18:41
f2898bd to
5b4357c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #629.
Three
intlimits stood between a large chunk and theu32segment length the format allows (up to 4 GiB):FsstEncodingEncoderkept the code offsets in anint[], overflowing once the compressed codes passed 2 GiB. They are along[]now, as Rust'sfsst_compressswitches toi64offsets pasti32::MAX.VortexWriterwrote a segment through oneByteBufferview, which cannot pass 2 GiB. It writes in 1 GiB slices, still zero-copy.SerializedArrayDecodernarrowed the segment length toint; it stayslong(bounds still enforced byIoBounds.checkRange).Test:
FsstLargeChunkInteropIntegrationTestwrites ~2.46 GB of slightly compressible binary under forced FSST, then vortex-jni and vortex-java read back the rows past 2 GiB; it fails with the originalinteger overflowwithout the fix. It needs several GB of memory, too much for CI runners, so it gets a new opt-inlargetag, excluded locally and in theciprofile (documented in AGENTS.md and the integration POM).Real data (Raincloud):
peoples-speech-clean-validation(2.3 GB audio column in one 8192-row block) now imports; its 6.35 GB CSV export is byte-identical to the vortex-jni-written file's.🤖 Generated with Claude Code