Repository navigation
fix: read and write string and binary chunks past 2 GiB in every encoding - #637
Merged
Merged
Conversation
…ding of the int limits on a chunk past 2 GiB: - VarBinOffsetArray read u32 offsets as signed ints, so a Rust-written varbin column with a data buffer between 2 and 4 GiB failed as out of range (mmmu). U32 offsets are read unsigned; an offset past the data buffer still fails as a VortexException. - VarBinEncodingEncoder refused a chunk past 2 GB of row bytes, which no byte[] holds contiguously. It copies the rows straight into its off-heap buffer, with i64 offsets. - VarBinViewEncodingEncoder summed its data in an int and wrapped. Its rows now fill data buffers up to Integer.MAX_VALUE bytes each (Rust's MAX_BUFFER_LEN), the views naming their buffer. - ZstdEncodingEncoder summed its length-prefixed payload in an int and wrapped (raw IndexOutOfBoundsException on a compact import of peoples-speech). The total is a long, and a varbin frame closes before 1 GiB: ZstdEncodingDecoder caps a frame at 2 GiB until ADR 0004's resource limits exist, so every frame this writer emits stays readable. - ZstdEncodingDecoder rebuilt varbin rows with i32 offsets; it takes i64 ones once the rows pass 2 GiB. Tests: the buffer and frame caps are injectable, so unit tests reach both splits with small caps (exact fill, overflow, inlined rows, empty, a row over the cap, values per frame combined with the byte cap). LargeChunkInteropIntegrationTest (the former FsstLargeChunk one, `large` tag) forces fsst, varbin, varbinview and zstd on ~2.46 GB of rows and has vortex-jni and vortex-java read the rows past 2 GiB; the three new cases fail without this change. Java writes i64 varbin offsets, so the unsigned u32 read is pinned by VarBinArrayTest and cross-checked only by Raincloud's mmmu, which now passes conformance; mmmu also imports, and a compact import of peoples-speech exports a byte-identical CSV. Rust's zstd writes 8192 values per frame where Java's compact writes one frame per array: #636. Closes #633 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011fRbYryTxfcu5j2CXwkSzd
dfa1
force-pushed
the
fix/633-large-chunks
branch
from
November 1, 2026 20:13
92687cf to
b7191dd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #633. Follow-up to #629, which covered FSST and the segment path.
Fixes (all found on real Raincloud files)
VarBinOffsetArrayreadsU32offsets unsigned — Rust-writtenvarbinwith a 2–4 GiB data buffer (mmmu) failed as out of range; an offset past the buffer still fails asVortexException.VarBinEncodingEncodercopies rows straight into its off-heap buffer (i64 offsets) when they don't fit onebyte[], instead of refusing the chunk.VarBinViewEncodingEncoderfills data buffers up toInteger.MAX_VALUEbytes each (Rust'sMAX_BUFFER_LEN), views naming their buffer, instead of wrapping aninttotal.ZstdEncodingEncodersums its payload in alongand closes a varbin frame before 1 GiB;ZstdEncodingDecoderkeeps its 2 GiB per-frame cap (a guard until ADR 0004's resource limits), so every frame Java writes stays readable. Was a rawIndexOutOfBoundsExceptionon a compact import ofpeoples-speech.ZstdEncodingDecoderrebuilds varbin rows with i64 offsets past 2 GiB.Tests
VarBinArrayTest$U32Offsets: unsigned read past 2^31, and a hostile offset past the buffer still rejected.LargeChunkInteropIntegrationTest(renamed fromFsstLargeChunkInteropIntegrationTest,largetag, opt-in): forces fsst/varbin/varbinview/zstd on ~2.46 GB of rows; vortex-jni and vortex-java read the rows past 2 GiB. The three new cases fail onmain.u32read is cross-checked only by Raincloud'smmmu(now passes conformance; a fixture that size can't be committed).Real data:
mmmupasses conformance and imports (CSV identical); a compact import ofpeoples-speech-clean-validationexports a byte-identical CSV.CHANGELOG: #629's line narrowed to FSST (it claimed every encoding), #633 added.
Reviewed independently (vortex-reviewer agent): no blockers; its should-fix items (CI coverage of the split logic, a negative u32 test, a test javadoc that overstated what the varbin case covers) are addressed in this commit. Rust's 8192-values-per-frame zstd framing is a separate parity gap: #636.
🤖 Generated with Claude Code
https://claude.ai/code/session_011fRbYryTxfcu5j2CXwkSzd