Repository navigation
Admit E8M0 block scales to the order-1 scale coder (includes the length-prefix fix) - #4
Merged
Merged
Conversation
rANS, tANS, and the arithmetic coders refused to decode any plane declaring more than 1 GiB, while their encoders wrote such planes without checking. A 2.1 GB BF16 embedding (128256 x 8192) therefore encoded and then failed to decode with "implausible orig_len". One rule in codecs::length_prefix now decides which declared lengths are accepted, and both sides apply it. Up to 1 GiB any ratio is accepted, as before. Above that, the encoded body must be at least 1/64 of the declared length, which still bounds what a short or corrupted payload can make the decoder allocate. rANS and tANS store a plane raw when the rule rejects it; the arithmetic coders, which have no raw form, return an encode error instead of an undecodable payload. The rANS and Huffman encoders also cast the plane length to u32 unchecked, so a plane over 4 GiB got a wrapped prefix. They now store such planes raw. Encoder output is unchanged for every plane up to 1 GiB.
Order1ScaleAC accepted only FP8 scale planes, on the assumption that MXFP4's E8M0 block scales have no row structure. They do: across gpt-oss-20b's 597M E8M0 scales the entropy conditioned on the previous scale in the row is 1.20 bits, against 1.89 bits unconditioned. E8M0 Scale and GlobalScale planes now reach the coder, and the per-plane trial encode still keeps whichever codec is smallest.
The two E8M0 scale-block tests asserted the old FP8-only gate. With E8M0 admitted, the effective menu includes Order1ScaleAC, which wins the synthetic NVFP4 fixture plane.
The pinned ruff flags both (SIM102, I001).
The Rust test, coverage, and lint jobs build pyo3-ffi against the runner default Python, now 3.14, which PyO3 0.24 does not support.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two commits.
1. Length-prefix limit shared by encoders and decoders (
fix(core), 090b9f9)rANS, tANS, and the arithmetic coders refused to decode any plane declaring more than 1 GiB, while their encoders wrote such planes without checking. A 2.1 GB BF16 embedding (128256 × 8192) encoded and then failed to decode with
implausible orig_len. Both public releases (1.0.0, 1.0.1) carry this.codecs::length_prefixdecides which declared lengths are accepted; encoders and decoders both apply it.as u32casts in the rANS and Huffman encoders no longer wrap planes over 4 GiB.2. E8M0 scales reach
Order1ScaleAC(feat(core), d6dec5b)The coder accepted only FP8 scale planes, on the assumption that MXFP4's E8M0 block scales have no row structure. Measured on gpt-oss-20b's 597M E8M0 scales: 1.20 bits conditioned on the previous scale in the row, against 1.89 bits unconditioned. E8M0 Scale and GlobalScale planes now reach the coder, and the per-plane trial encode still keeps the smallest codec. The README dtype table is updated.
Synthetic check: on an 8.3 MB row-correlated E8M0 plane, the container is 1.54 MB with the default menu against 2.41 MB with the generic coders only, and decodes byte-exact.
Known issue, not addressed here: on a small plane (46 KB) the default menu picked
Order1ScaleACalthough its container came out larger (31,350 B against 16,955 B with the generic coders). The trial encode appears not to charge shared-state table bytes to the plane.Tests
cargo test --workspace: 916 passed, 2 ignored (the two above-1 GiB round trips, run separately in release mode and passing).cargo clippy -p ptwm-core --all-targets -D warnings: clean.