Cut fixed per-container overhead with compact formats; fix cross-version decode and reader hardening by khwstolle · Pull Request #6 · khwstolle/ptwm

khwstolle · 2026-10-28T12:52:13Z

Summary

Resolves #5. Single-tensor output carried about 2.7 KB of fixed bytes, so small tensors came out larger than their input. This PR adds a compact single-tensor frame (\x89PTWS, the new Compressor.compress default) and a compact multi-tensor container (\x89PTWC, the new default for multi-tensor writers). It also fixes defects found on the way. The most serious: containers written by any other release did not decode, because built-in ids embed the crate version.

Output	64 B constant bf16	(64, 1024) tiled bf16 ratio	4 KB random fp32
default, one tensor (`\x89PTWS`)	44 B	86.1	3,501 B
compact container (`\x89PTWC`)	103 B	82.2	3,585 B
legacy container (`\x89PTWM`) / v1.1.0	2,503 B	30.5	6,318 B
zstd-19	—	—	3,827 B

All five acceptance criteria of #5 hold. benchmarks/container_overhead.py reports the sweep, and tests/ptwm/benchmarks/test_container_overhead.py enforces the limits in CI.

Format changes

\x89PTWS compact frame: varint header, a compact chain encoding (CW2) with the Source shape bound to the header, and per-plane codec records. It stores the raw bytes when no codec helps.
- Fixed cost is at most 50 B for bf16 and 62 B for fp32.
- Falls back to \x89PTWC for streaming or shared-state planes.
\x89PTWC compact container:
- Built-in table entries take 1–3 B instead of about 125 B.
- Records carry no names, and the index holds the names.
- Planes have no stored codec id.
- A metadata xxh64 covers the header and tables.
- Parsing is strict: reserved bits must be zero, varints must be minimal, and every section must be consumed exactly.
- 64 small fp32 tensors cost 83 B per tensor (legacy: 235 B).
Legacy \x89PTWM stays writable through container_layout="legacy" (plus Framing.CONTAINER for a single tensor) and --container-layout legacy on the CLI.
- It writes the full v1.1.0 extension table with v1.1.0 built-in ids.
- v1.1 reads these files, including exploded LMDB and WebDataset stores. This was checked against the real v1.1.0 code.
Transcode: a new "ptwm-manifest-c" version-2 manifest for PTWC. The legacy manifest keeps working byte-exactly, including for files written with other releases' ids. implode_ptwm rejects unknown formats, versions and fields.

Existing defects fixed

Cross-version decode: built-in ids from any release normalise to the current ones. Opening a file that uses only built-ins skips the discovery, keyring and install pass.
Reader sizing: plane sizes come from a symbolic pass over the chain, not from running it on a zero buffer of the declared size. Declared sizes are bounded by the record size.
Allocation from file-declared sizes: allocations that file-declared sizes control are capped. A 53-byte crafted frame used to abort the process.
Format.BYTE Source sizing: the chain now declares the real element count. Old files that over-declare still decode. The writer rejects chains whose Source size is wrong.
Hardening:
- checked casts and overflow checks in metadata, inspect and the record writers;
- bounded section parsing;
- duplicate and renamed tensor names rejected;
- the chain-count limit enforced;
- skip_missing honoured;
- the chain-blob consumed length taken from the cursor.
Python:
- Delta compression on TORCH/NUMPY input crashed; fixed.
- float64 and int64 NumPy arrays did not decode; fixed.
- Safetensors files without metadata lost their compressed-tensor record; fixed.
- framing and container_layout are coerced from strings.
- New ContainerHandle opens a container once and avoids copying the blob on every call; it borrows only immutable buffers.
Docs: docs/architecture/wire-format.md is rewritten against the code; it had wrong sentinel bytes, plane fields and alignment claims. New release notes are in docs/site/content/70.release-notes.md.
Fuzzing: new fuzz targets for the compact formats.

Checklist

cargo fmt --check and cargo clippy --all-features -- -D warnings pass locally (also with --all-targets)
cargo test --workspace and uv run pytest pass locally (Rust 1,112 passed; Python 1,055 passed, 12 skipped)
User-visible changes are reflected in the docs site
New behaviour has a round-trip / regression test where applicable

Notes for reviewers

Default outputs need v1.2 or later. Older readers reject \x89PTWS and \x89PTWC with an error and never misparse them. The release notes say how to write v1.1-readable files.
The legacy layout keeps the full v1.1.0 extension table on purpose. v1.1's transcoder rebuilds that table for exploded stores, so a shortened table broke v1.1 readers. The size savings are in the new formats only.
Changed tests: three tests that asserted a shortened legacy table now assert the full v1.1 table. Two laziness tests now spy on ContainerHandle.decode. No test was removed.
Golden test files: crates/ptwm-core/tests/fixtures/ holds files from the legacy writer, including one with v1.0.0 ids and a byte-input file that over-declares its size.
Known limit: Order1ScaleAC decode output is bounded only by the plane limit (16 × declared size + 4 KiB). A crafted file can make it allocate a lot, but it no longer aborts.
History: the branch has 32 commits, including four wip(...) snapshots taken to survive container restarts. Squash-merge is recommended.

Generated by Claude Code

Single-tensor Compressor.compress output carried about 2.7 KB of fixed bytes, so small tensors came out larger than their input (issue #5). - The container writer interns only the plane codecs that the planes reference, not all 16 built-in codecs. A single-tensor container drops from about 2.5 KB to 0.5-1 KB of non-payload bytes. Readers resolve codec indices by position, so existing readers decode the new files. The explode/implode transcoder rebuilds whichever table layout matches the recorded hash, using the archive-wide codec list so a one-tensor implode keeps its codec indices. - New compact single-tensor frame (magic \x89PTWS): varint header, optional shape/hash/CRC/delta fields, the chain, and per-plane codec records. It stores the raw bytes when no codec helps. Fixed cost is 10-130 B. Compressor uses it by default (Framing.COMPACT) and falls back to the container for streaming or shared-state planes. Framing.CONTAINER keeps the old layout; the CLI file writers and the ablation harness pin it. The decoder bounds the chain's declared size by orig_size before the descriptor pass, so a corrupt frame cannot request gigabytes. - CompressionConfig exposes emit_payload_hash and emit_plane_crc. - benchmarks/container_overhead.py reports the size sweep; a CI test enforces the issue's acceptance limits. New compact_decode fuzz target. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

Pin the \x89PTWM wire format before the format work for issue #5. - tests/fixtures/legacy_v1_1_0.ptwm: multi-tensor container with the v1.1.0 Extension Table layout (all 16 built-in codecs, written with ContainerWriter::set_chain_registry). Holds bf16, fp16, fp32, fp8 and int8 tensors, two scale tensors that share an Order1ScaleAC prelude state, a chunked plane, an inline chain and a dependency. - legacy_v1_0_0.ptwm: the same file with built-in ids and labels re-derived for version 1.0.0 and the table hash fixed up. Its decode and transcode tests are ignored until cross-version built-ins land. - legacy_trimmed.ptwm: output of the current trimmed-table writer. - legacy_p4_byte_fp32.ptwm: fp32 from BYTE input whose Source declares four times the real size. tests/legacy_fixtures.rs checks that each fixture decodes to the expected (length, xxh64) values and that implode(explode(x)) == x. It also pins the blake3 hash of compress_model output for a fixed input, so later changes to legacy bytes are caught. An ignored test regenerates the files (see the module docs). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

- Add legacy_p4_byte_fp32_v1_1_0.ptwm: the P4 tensor in the v1.1.0 layout (full Extension Table). It is byte-identical to v1.1.0 compress_model output. legacy_p4_byte_fp32.ptwm is now documented as current-writer (trimmed table) output. - Add legacy_v1_0_0_native.ptwm, written once by the v1.0.0 release (a17d403). Reword the legacy_v1_0_0.ptwm docs: it is the v1.1.0 fixture with ids re-derived for 1.0.0, not v1.0.0 writer output. - rederive_builtins parses the header by hand instead of calling ContainerReader::open_partial, so it works on blobs with built-in ids of other versions. Document the equal-length version limit. - Test explode->implode and decode on in-memory main_container() output (trimmed table plus shared Order1ScaleAC state). - Fix clippy cloned_ref_to_slice_refs in transcode tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

Stage 2 of the issue #5 work (synthesis E1-E8, E11). Legacy writer output is unchanged; only new error paths are added. - metadata: the tstr bounds check uses checked arithmetic, and the shape array reservation is capped by the bytes left (E1, E2). - inspect: list_extension_table_entries checks offset + length for overflow instead of panicking (E3, bounds part only). - tensor_record / plane_record writers: lengths and counts that do not fit their wire integer return InvalidContainer before any byte is written, instead of being truncated (E4). - ChainRegistry::write rejects chain ids above 0x7FFF and more than 0x8000 chains; compress_model's registry builder fails past 0x8000 distinct chains (E5). - ContainerWriter rejects duplicate tensor names. get_tensor_record_bytes scans the whole run of equal name hashes and compares record names, so a hash collision resolves to the right tensor (E6). - The prelude and the chain registry are parsed only within shared_prelude_length (E7). - read_chain_blob and chain_blob_info take the table size from the parse position (new ExtensionTable::from_bytes_prefix), not from a re-serialization (E8). - decode_tensor_shape and list_plane_summaries take skip_missing, in the bindings and the Python wrappers (E11). Each fix has a regression test (Rust unit tests and tests/ptwm/wire_format/test_legacy_hardening.py). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

- write_tensor_record truncates out to its starting length on any error, so inline-chain and terminal-record failures no longer leave a partial record behind; regression test with a non-empty prefix buffer. - ContainerReader: drop the unreachable shared_prelude_length == 0 fallback for tensors_offset; test that a zero-length section is rejected and that tensors_offset is the section end. - Python test pins chain_blob_info returning the parsed Extension Table size (padded capability blob moves it by one byte). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

Stage 3 of the issue #5 work (synthesis X1, E9, E12; legacy manifest parts of section 6). Legacy writer output is unchanged. - extension::builtin_registry: a OnceLock map over every built-in, so dispatch_builtin, is_builtin and builtin_canonical_id are O(1). OpId::ALL and OpId::builtin_name replace the per-call id derivation. - normalize_builtin_ids maps entries labelled io.ptwm.builtin.<name>@<ver> to the current ids, but only when <name> has a built-in entry and the id equals derive(BUILTIN_PUBKEY, name, ver). Forged entries stay non-built-in. It runs after the blake3 check in ContainerReader, on legacy inline-chain tables and in read_chain_blob / chain_blob_info, so older chain-blob caches load. ContainerReader exposes the table's builtin_version. - E9: when every Extension Table entry is a built-in, the reader skips discovery, the keyring and apply_fresh_install. An integration test with temporary HOME/XDG directories checks that no file is created. - Writers take a built-in version: build_extension_table(.., version), ExtensionTableBuilder::with_builtin_version / intern_builtin, ContainerWriter::set_builtin_version (file-level and inline tables), write_tensor_record_with_builtin_version. - New ContainerWriter::set_raw_extension_table (validated by encode_registry_for_table: canonical bytes, no trailing bytes, no embedded WASM, every chain op present), written verbatim. - Transcode: optional container_flags.builtin_version and registry.extension_table_b64 (omitted when unused, so existing manifests are unchanged). Explode records the version, carries the raw table only when no builder reproduces it, refuses inline chains of two different versions, and checks that the shared-prelude section re-encodes byte-exactly. Implode tries the trimmed table, then all built-ins, at that version, then the raw bytes, and errors when none matches the recorded hash instead of falling back silently. - E12: implode checks (ptwm_format, manifest_version) before parsing; every manifest struct denies unknown fields. - The v1.0.0 fixture tests are no longer ignored; the v1.0.0 fixtures decode and explode->implode byte-exactly. New tests cover mixed-version tables, forged labels, single-tensor implode of v1.0.0 files, manifest rejection, and old-id chain blobs from Python. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

- explode re-encodes every inline-chain record with the built-in version implode will use and refuses the container when the bytes differ. An inline Extension Table that mixes built-in versions was accepted before and imploded to other bytes. - One built-in version rule, extension::is_valid_builtin_version (non-empty, at most 64 bytes, no '@'), now applies in normalize_builtin_ids, builtin_entry_at_version, ExtensionTableBuilder::with_builtin_version (now fallible, new ExtensionError::InvalidBuiltinVersion), build_extension_table and ContainerWriter::set_builtin_version. Entries with an invalid version stay non-built-in, so explode never records a version implode rejects. - ptwm-py list_extension_table_entries normalises built-in ids, so ptwm.ablation compare/attribute match built-ins across releases. Tests: explode_rejects_inline_table_with_mixed_versions, invalid_builtin_versions_are_not_recorded, invalid_versions_are_not_normalised, with_builtin_version_rejects_invalid_versions, a build_extension_table check in set_builtin_version_must_come_first, and the Python test_list_extension_table_entries_reports_current_builtin_ids. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

Stage 4 of the issue #5 work (synthesis section 3, P4 and P5). P5, decode side: - chain::runtime::propagate_terminal_descriptors(_bounded) walks propagate_descriptors in topological order with no plane buffers. It mirrors forward_chain: inputs sorted by dst_input_idx, edge role overrides, terminal role re-tagging (fused kernels emit the same descriptors). The bounded variant rejects the Source, and any node output, above max_plane_len(orig_size) = 16 x orig_size + 4096. chain_terminal_descriptors now delegates to it; chain_source_descriptor is shared by all callers. - Source::output_descriptor uses checked arithmetic (no overflow panic or wrap on 255 x u32::MAX dims). - validate_chain looks edge sources up with .get() (no index panic). - Legacy container decode: Source <= 8 x orig_size (P4 files stay readable), chunk-table decoded sizes and their sum bounded before decode, decoded planes bounded, inverse output must equal orig_size. - Compact frame decode: Source must equal orig_size; each plane must decode to its descriptor length. P4, encode side: - Python build_chain_blobs/source_shape_for in preprocessing/_chains.py give each builder a Source shape in units of its own Source dtype that covers exactly the raw byte count, skipping builders that cannot, and fall back to a byte passthrough over all bytes. Used by core/_compressor.py, sharding/_writer.py and integrations/_compress_safetensors.py. - Compressor reads dtype and shape before delta encoding, fixing the AttributeError for delta + TORCH/NUMPY input. - compress_model skips candidate chains whose Source size differs from the raw length and errors when none is left. - The forward pass takes the source element width and nibble packing from the chain's Source (chain-internal dtype codes) instead of the record's public dtype code, so encode and decode descriptors agree. Legacy golden output and measured sizes are unchanged. - New binding _core.chain_source_byte_len. Tests: symbolic == forward reference on every production/explorer chain blob (new fixture production_chains.bin, kept current by test_production_chain_fixture.py) and on 3000 random chains; bounded walk, delta-op, missing-plane and overflow cases; container 4 TiB/4 GiB Source fail fast, 8x tolerance, understated Source, orig_size mismatch, chunk-table bound; compact exact-Source checks; writer skip of mis-sized candidates; Python test_source_shape.py. The P4 legacy fixture generator now writes the exact chain and patches the stored Source dim, which reproduces the pinned v1.1 bytes (fixture files unchanged). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

Saved so a container restart cannot lose it. The resumed Stage 4 review and fix validate these changes and follow up in the next commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

Bound codec output while decoding (spec section 3, P5: no buffer beyond orig_size): - PlaneCodec::decode_bounded takes a hard max_len. The default decodes and checks the length. Zstd and ZstdDict decode through a streaming reader that stops one byte past the bound (ZstdDict with a dictionary caps its bulk capacity). rANS, tANS, the arithmetic coders, Order1Arithmetic, ContextMixingLite and NeuralPredictor reject a length prefix above the bound before the decode loop. - Compact frames pass the exact plane length, legacy whole planes pass max_plane_len(orig_size), chunks pass the table's decoded size. A 98 KB frame of 3 GiB zeros now fails at 4 KiB instead of growing the process by 3 GiB. - New tests: per-codec bound tests, a hand-built Zstd RLE bomb, and tests/decode_bomb.rs, which plants a 512 MiB bomb in compact, legacy and chunked files and checks the largest allocation stays under 64 MiB. Other findings: - The spherical and alpha-stable builders decline shapes with an empty dimension, so empty 2-D float tensors such as (5, 0) compress again. - max_plane_len documents why 16x holds: the 8x overstated Source and the 4x alpha-stable scale plane never occur in the same chain. - test_chains.py uses the CHAIN_BF16_MICROSCALE_REPACK import (lint). Production table change carried by the previous commit (5aa2cce): CHAIN_BF16_MICROSCALE_REPACK is no longer a bf16 production candidate. With the encoder taking the element width from the chain's Source, the chain stopped failing and would win trial-encode on some bf16 tensors, changing the legacy writer's output. Removing it keeps that output byte-identical; the builder still works when passed explicitly. The stage report of ce3e944 did not mention this. Legacy and compact output for the reviewer's typed tensor set is unchanged by this commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

Stage 5 of the issue #5 work (synthesis section 2 CW2 and PTWS, P3). - crate::varint: canonical unsigned LEB128 (minimal encoding, at most 10 bytes, fits u64) and a bounds-checked Cursor whose len_varint never exceeds the remaining bytes. The compact frame's private helpers moved here. - chain::wire_compact (CW2): varint counts, op_ref resolved by the caller (OpId wire id for PTWS, Extension Table index for PTWC later), spine-edge compression (leading (i,0)->(i+1,0) edges are a count), one-byte terminal refs, one-byte compact roles with a 0xFF escape, and a bound Source (param_len 0 = params of the tensor's own shape and dtype). Every field has one encoding; the reader rejects the others (overlong varints, needless escapes, explicit spine edges, explicit Source params equal to the binding, unknown etag bits or role bytes). - transforms::source: chain_dtype_for_public (mirror of _CHAIN_DTYPE) and bound_source_params. - PTWS frame, changed in place (unreleased): CW2 chain replaces the op table and registry chain wire; plane head is one varint codec_id << 2 | state with state 2 = inline, version 0, empty (state 1 with an empty version-0 state is rejected); the last plane's payload_len is omitted; orig_size is stored only when the shape does not give it (new FLAG_SIZE, rejected when redundant); a stored dtype name equal to the canonical one is rejected. Shapes above 64 dims fall back to the full container instead of writing an unreadable frame. - Fixed overhead with hash and CRCs on: fp32 random 125 -> 68 B, bf16 random 98 -> 48 B, fp32 constant 124 -> 62 B. - Fuzz: compact_decode follows the new header order; new target chain_wire_compact asserts accepted chains re-encode to the same bytes. - tests/ptwm/benchmarks/test_container_overhead.py bounds compact non-payload bytes per dtype (fp32 <= 70 B, bf16 <= 55 B) for random and constant tensors at every size. - wire-format.md and the Framing docstring describe the new frame. Tests: varint (round trip, non-minimal, overflow, cursor bounds); CW2 round trip on all 500+ production/explorer chains in the fixture (explicit and bound), exact bytes of the fp32 split chain (19 B), edge cases (non-spine edges, vendor bytes, escaped refs and roles, large op refs, 0-d Source, 255 nodes), unencodable chains, canonicality rejections, and 60k random mutations that must be rejected or re-encode identically; PTWS bound Source, shape-independent chain bytes, bound Source without a frame shape, derived vs stored orig_size, header and plane-state canonicality, last-plane truncation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

- PTWS frames are self-delimiting again: every plane, including the last, stores its payload length, and the reader rejects bytes appended to a coded or stored frame. - The compact writer stores plane CRCs only without the payload hash, which already covers every plane of the one tensor; a reader rejects a frame with both flags. This pays for the restored length: default fp32 overhead is now at most 55 B (split chain) and 61 B (IntDelta chain). - Docs (compact.rs, varint.rs, wire-format.md, _config.py) now claim one encoding per field, not per tensor, and name the remaining writer choices (stored vs coded body, plane state 0 vs 2). - Overhead tests: tighter random/constant bounds and a smooth-data case (IntDelta chains) held to 70 B. - Fuzz: seed corpora for compact_decode and chain_wire_compact; compact_decode asserts that no accepted frame survives an appended byte. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

fuzz/corpus/ is gitignored, so the seeds the previous commit's fuzz targets point to were left out. Force-add them, as for container_open. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

Partial work on the compact multi-tensor container, saved so a container restart cannot lose it. May not compile yet; the resumed Stage 6 implementer completes it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

Stage 6 of the issue #5 work (synthesis sections 1, 2, 4, 5). Completes the interrupted snapshot b7c3dce. - Magic \x89PTWC\r\n\x1a\n. Header: hflags u16 (all bits reserved), varint lengths of the Extension Table, prelude and registry, and meta_xxh64 over the header and those sections. Footer: index_len u64 and the unchanged EOF sentinel. Each section parser gets exactly its slice and must consume it (crate::ptwc). - Compact Extension Table (extension::compact_table): tags 0x01 op id, 0x02 codec id, 0x03 built-in name, 0x7F full third-party entry. Built-ins expand to the reader's own entries, so dispatch, trust and the missing-flavor check are shared with the legacy reader. - Registry: CW2 chains whose op_ref is a table index. A Source with empty params in memory is a bound Source; the record's shape supplies its params. - Records (tensor_record_compact): no name, tflags/pflags with every reserved bit rejected, shape stored as varints (dtype name only when not canonical, otherwise raw metadata), inline chains with a first-use-order table, plane codec_id derived from the table entry. The writer rejects records whose stored fields disagree with the derived ones, so parse-then-write is the identity. - Index: names in body order; lookup compares names; duplicates, gaps and overlaps are rejected. - ContainerLayout {Legacy, Compact}; ContainerWriter::with_layout (new() stays Legacy); RecordEncoder / encode_record for parallel record serialization; set_compact_tables for verbatim tables. - CompressorOptions now derives Default and has container_layout (default Legacy until the stage that flips defaults). In Compact, compress_model binds Sources, so tensors that differ only in shape share one registry chain. Registry dedup now uses a HashMap for both layouts (Chain and Role derive Hash); legacy output is unchanged (the golden writer hash test passes). - Reader sniffs the magic and stores the layout; parse_record, tensor_record and tensor_records are used by tensor_shape, tensor_names (index only for PTWC), decode_tensor, decode_model and the ptwm-py info, delta-hash and plane-summary functions. The compact reader requires exact sizes (Source == orig_size, plane output == descriptor length). - Transcode explode rejects PTWC explicitly until the compact manifest lands. Tests: legacy fixtures rewritten as PTWC (registry and inline chains, shared prelude state, chunked planes, dependency, Rows layout, bf16/fp16/fp32/fp8/int8) decode identically and rewrite byte-exactly; compress_model matrix over both layouts x hash/CRC x chunked, with 0-d, non-canonical dtype name and dependency tensors; raw metadata, external state and state_extra records; canonicality rejection for header, table, record, plane, registry and index; bounded counts; a P4 file with Source slack fails in PTWC; Header::from_bytes on PTWC gives "bad magic"; bound Source gives one chain for 64 shapes; duplicate names; 30k record mutations re-encode identically or fail. Overhead for 64 fp32 (64,) tensors: 81 B/tensor with hash and CRC (legacy 229), 57 B without; built-in-only table at most 4 B/entry. Fuzz: new targets parse_wire_compact and extension_table_compact (re-encode identity), container_open also runs decode_model; PTWC seeds added. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

- tensor_record_compact: document that the in-memory `flags` byte is recomputed, as the legacy writer does, instead of claiming the writer rejects mismatches; pin it with a unit test. - ptwc: document the registry limit (chain ids 0..=0x7FFF, so at most 0x8000 chains, the legacy ChainRegistry limit) as a deliberate reading of synthesis section 2. - tests/ptwc.rs: single-tensor overhead bound (<= 160 B), a 4 GiB bound Source in PTWC fails fast, BYTE-input tensor, TF_RAW_META metadata and an External state plane round-trip through written PTWC containers. - fuzz workflow: add chain_wire, chain_wire_compact, extension_table_compact and parse_wire_compact to the full sweep. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

Stage 7 of the issue #5 work (synthesis sections 6 and 10 step 7). - explode of a \x89PTWC container gives a "ptwm-manifest-c" version 2 registry manifest: container_flags {embedded_wasm: false, layout: "compact"}, registry.extension_table as a list of {"op"} / {"codec"} / {"named"} / {"full_b64"} entries in table order, shared_state, and chains in registry order with a bound Source as params_b64 null. No registry_hash_blake3. Tensors come in body order; tensor and plane manifests have the legacy form (codec_table_idx = codec_ref, codec is informational and checked by the writer). - implode dispatches on (ptwm_format, manifest_version): ("ptwm-manifest", 1) takes the legacy path unchanged, ("ptwm-manifest-c", 2) rebuilds the table through the compact table reader (unknown ids and names, duplicates, built-ins stored in full are rejected) and writes table, registry and order verbatim with ContainerWriter::with_layout(Compact) and set_compact_tables. Any other pair is rejected. The compact manifest structs use deny_unknown_fields. - Legacy explode/implode output is unchanged (shared helpers only). - ptwm-py compress_model takes a keyword-only container_layout ("legacy" default, "compact"; other values raise ValueError), so Python can write PTWC now. compress_ptwm_blob, write_ptwm_shard, write_lmdb, LmdbWeightStore.put_tensor and write_webdataset pass a keyword-only container_layout through (default "legacy" until the default flips). - stores/_common.py: manifest_container_layout() and ExplodedModel.container_layout; docs state that per-tensor implode uses the archive-wide tables. Tests (Rust, tests/ptwc.rs): every stage-6 PTWC case (legacy fixtures rewritten as PTWC, compress_model matrix over both layouts x hash/CRC x chunked, bound Source, BYTE input, raw CBOR metadata, external state, P4 slack) now also checks implode(explode(x)) == x, the manifest form, body order and a one-tensor implode per tensor that decodes like the full file. New: manifest lists the table and binds Sources; named and third-party entries round-trip; malformed compact manifests are rejected (version/format mismatch, layout, unknown fields, entry kinds, unknown ids/names, duplicates, built-in in full_b64, chain id != position, codec disagreeing with the table); a frozen v1.1 implode gate and a ("ptwm-manifest", 1)-only gate both reject compact manifests. Tests (Python, tests/ptwm/stores/test_compact_layout.py): layout keyword on compress_ptwm_blob/write_ptwm_shard, unknown layout rejected, compact manifest form and identity per method hint, Rust JCS == Python JCS, per-tensor raw_from_members in both layouts, LMDB blob and exploded stores, put_tensor (incl. delta) in either layout, WebDataset single and multi-shard stores with a compact registry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

- compress_model docstring: an unknown container_layout raises ptwm.Error (translate_errors wraps the binding's ValueError), not ValueError. - LMDB: write_lmdb(container_layout="compact") records the layout in the \x00fmt record; LmdbWeightStore.put_tensor now defaults container_layout to None and uses the store's recorded layout ("legacy" when none is recorded). The fmt record of a legacy store is unchanged. - Tests: put_tensor inherits the store layout (legacy and compact, incl. delta and reopen); compress_model error type for an unknown layout. test_lmdb_put_tensor_in_either_layout now passes container_layout="legacy" explicitly, because the default it pinned changed deliberately. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

Partial Python and binding work (ContainerHandle, _magic, layout plumbing), saved so a container restart cannot lose it. May not build yet; the resumed Stage 8 implementer completes it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

Stage 8 of the PTWC work (synthesis sections 3 E3/E10/E14 and 7). Bindings (ptwm-py): - ContainerHandle pyclass: one opened ContainerReader over a borrowed read-only buffer (bytes, read-only memoryview or mmap); writable buffers are copied once. Methods: names, decode, decode_model, info, delta_hash, shape, plane_summaries, plus layout and zero_copy. - decode_tensor, decode_model, decode_tensor_info, decode_tensor_delta_hash, list_tensor_names, decode_tensor_shape and list_plane_summaries are thin wrappers over a handle and take any buffer instead of copying it; all honour skip_missing. - _core.container_layout(blob) maps the magic to "legacy"/"compact". - list_extension_table_entries reads both layouts through the new core read_extension_table (bounds-checked region, verified hash, compact entries expanded). Core: - read_extension_table(buf) dispatches on the magic. - tensor_names and decode_model return body (write) order in both layouts (E14). Python: - ptwm._magic: MAGIC_* constants, container_kind, is_container; used by the Decompressor, TensorIndex and stores.detect_format. - ContainerLayout enum (string coercion) and CompressionConfig.container_layout (default stays LEGACY until the default switch); container_layout threaded through the Compressor, compress_ptwm_blob / write_ptwm_shard, the LMDB and WebDataset writers and compress_safetensors_file; the enum is accepted everywhere. - ContainerHandle used by TensorIndex, Decompressor.decompress/decompress_container_to_state_dict, utils.decode_ptwm_tensor, the HF loader and SafeOpen. - Framing docstring: only LEGACY container output is v1.1-readable. Tests: tests/ptwm/core/test_container_handle.py (new); handle skip_missing and inspect tests in test_legacy_hardening.py; layout forwarding in test_compress_safetensors.py; enum test in test_compact_layout.py; store tests check magics through _magic. The two SafeOpen laziness tests now spy on ContainerHandle.decode, because SafeOpen no longer calls _rust.decode_tensor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

- ContainerHandle borrows a buffer only when its owner is immutable: an exact bytes object or an mmap opened with ACCESS_READ, directly or through memoryview links. Read-only views of mutable memory (memoryview(bytearray).toreadonly(), numpy arrays with writeable=False) are now copied, because the owner could still write them while the reader runs with the GIL released. The free container functions use the same rule. SAFETY comments and docstrings corrected. - compress_ptwm_blob, write_ptwm_shard and write_webdataset validate container_layout first and raise ValueError, before any file or directory is created. - test_lmdb pins the legacy magic of the default store layout again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

Stage 9 of the PTWC work (synthesis section 10 step 9). Defaults: - Rust: ContainerLayout::default() is Compact, so CompressorOptions::default() writes \x89PTWC. ContainerWriter::new still writes Legacy. - ptwm-py compress_model: container_layout defaults to "compact". - Python: CompressionConfig.container_layout defaults to COMPACT; _rust.compress_model, compress_ptwm_blob, write_ptwm_shard, write_lmdb, write_webdataset and integrations.compress_safetensors_file default to "compact". LmdbWeightStore.put_tensor still uses the store's recorded layout, and "legacy" for a store that records none. - Single-tensor Compressor.compress keeps the \x89PTWS frame by default and falls back to a PTWC container; with container_layout=LEGACY the fallback is \x89PTWM. Framing.CONTAINER writes the configured layout, so CONTAINER + COMPACT (the default) gives PTWC. Documented in the Framing, ContainerLayout and CompressionConfig docstrings. CLI: `ptwm compress` and the standalone compress entry point take --container-layout {compact,legacy} (default compact). It reaches raw-file, delta, per-tensor safetensors, directory, mode a/b, LMDB and WebDataset writers. The random-access, store and inspect tools open both layouts. cli/README.md documents the flag. Tests pinned to LEGACY because they read legacy bytes (deliberate default change): compressor/container unit tests that parse legacy records or patch legacy offsets, legacy_fixtures main_options (golden legacy hash), test_extension_table.py, test_v1_header.py, test_legacy_hardening.py _legacy_blob, test_transcoder registry_hash test (legacy manifest only), the legacy container floor test in test_container_overhead.py (benchmark measure/compress_checked take a layout). Tests updated for the new default: ptwc.rs legacy_layout_stays_the_default became compact_layout_is_the_default; test_compression_config default is COMPACT; test_compact_frame container/streaming fallback, test_zstd outputs and test_lmdb blob keys now cover default, compact and legacy; decode_bomb container tests run both layouts. New: test_v1_header compact default test, test_new_writes_default_to_the_compact_layout, tests/ptwm/cli/test_container_layout_flag.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

- Docs: legacy (\x89PTWM) output is readable by v1.1 and later, not by every release; the legacy writer is byte-identical to the current trimmed-table writer, not to the v1.1.0 writer. - CLI --delta passes the reference file's bytes instead of its path (it always failed with TypeError) and honours --dtype. - CLI --mode b --out F writes F (several shards go to a directory named after F without '.safetensors') instead of leaving a staging directory. - Directory compression runs in-process for one worker and uses spawned children for several; a forked child could deadlock on locks held by the parent's Rust or torch threads. - Tests: cross-release chain-blob test covers both layouts; new CLI tests for --delta, WebDataset, mode a/b, directory and multi-process runs in both layouts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

Stage 10 of the issue #5 work (synthesis section 3 E13/E15, section 10 step 10). Docs: - docs/architecture/wire-format.md rewritten against the code: correct EOF sentinel bytes (89 54 57 4D 45 4F 46 1A), chain registry inside the shared-prelude section, the magic rule runs old-reader-rejects-new, real legacy plane record fields (no per-plane shape or hash; codec is dispatched by codec_table_idx), no payload alignment. New sections: conventions, versioning (built-in semantics keyed by wire id; a semantic change needs a new OpId/CodecId), legacy Extension Table, built-in ids, prelude, registry and chain wire, tensor record, index, what the legacy reader checks, PTWC (header, table, registry, record, plane, index, footer, measured sizes), CW2, PTWS, compatibility and reader limits. - Site: concepts/5.container.md (both layouts, compatibility, correct hash and error claims), guides/02.python-api.md (container_layout knob, pre-v1.2 readers), guides/01.cli.md (--container-layout), new 70.release-notes.md: default output needs >= v1.2 readers, how to write v1.1-readable files and the built-in id caveat. GLOSSARY and the architecture README updated. Benchmark (E15): - container_overhead counts plane payloads, inline state and shared prelude state as payload for every format (via explode_ptwm), reports ptws, ptwc and ptwm rows, labels rows by the magic written, and adds a multi-tensor per-tensor-overhead sweep over both layouts. - Tests: codec data counted alike in all three formats, format labels, single-tensor PTWC bound (<= 160 B), multi-tensor per-tensor bounds (<= 95/100/70 B and under half the legacy cost). - benchmarks/README.md updated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

- CLI: `ptwm compress model.safetensors --container-layout legacy` now writes every tensor of the per-tensor shell as a full \x89PTWM container (Framing.CONTAINER) instead of a \x89PTWS frame. - Legacy writers stamp built-in ids and labels of LEGACY_BUILTIN_VERSION ("1.1.0") instead of the crate version, so legacy output stays readable by v1.1 after a version bump. The transcoder manifest omits builtin_version when it equals that constant. - benchmarks: codec_data_bytes read shared state from the wrong manifest key and never counted it; tests now cover inline and shared state. - Bound the Extension Table, capability blob and dependency allocations by the input; reword the reader-limits section to match. - Docs: correct the CLI guide, release notes, wire-format.md and 5.container.md (mode a is the safetensors default). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

zstd_decode_bounded reserved min(plane length, plane bound) bytes before decoding. Both come from the file, so a 53-byte frame with an edited shape asked for 32 GiB and the process aborted in handle_alloc_error. The first reservation is now also capped by what the payload can decode to (32 KiB per compressed byte) and made with try_reserve; the buffer grows only as data decodes. zstd_dict decodes with a dictionary through the same bounded stream reader instead of a bulk buffer of the declared size, and Order1ScaleAC grows its output in chunks with try_reserve. The legacy reader now checks at open that every record name hashes to its index entry and that names are unique. A bit flip in a record name used to rename the tensor silently, giving two tensors one name. The container reader names a \x89PTWS frame instead of reporting it as truncated (it has no EOF sentinel). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

The legacy writer interned only the codecs its planes used. Exploded LMDB and WebDataset stores of such a file carry "ptwm-manifest" v1 manifests whose codec_table_idx values index that short table. The v1.1 transcoder rebuilds the table with all 16 built-in codecs and copies the indices unchanged, so v1.1 decoded those stores with the wrong codecs ("bad huffman tag", rANS frequency errors). compress_model now writes the full v1.1.0 table for the legacy layout and trims it only in the compact layout. The legacy fixtures stay as they are; legacy_trimmed.ptwm and legacy_p4_byte_fp32.ptwm are now built by rewriting writer output with a trimmed table, and a new test checks that legacy writer output equals its own v1.1.0-layout rewrite. A Python test checks every plane's codec_table_idx in exploded legacy manifests against the table the v1.1 transcoder rebuilds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

DType.from_dtype compared np.dtype instances with `in`, which uses numpy's coercing __eq__: np.dtype("float64") == None and np.dtype("int64") == int are both true. float64 arrays mapped to NONE (code 0) and int64 arrays to INT32, so Compressor(Format.NUMPY) wrote frames that Decompressor could not rebuild. np.dtype inputs are now compared as dtypes, then by name. Compressor.compress raises InvalidDTypeError for a TORCH or NUMPY input whose dtype has no code, instead of writing code 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

…data set_compressed_tensors_metadata wrote the ptwm_compressed_vectors entry only into a non-empty metadata dict. A source saved without metadata (f.metadata() is None) lost the entry, so readers and `ptwm decompress` returned each compressed tensor as its raw uint8 frame. The function now creates the dict when it is None and returns it, and decompress writes no metadata when none is left. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

framing="compact" (or "COMPACT", or 1) stayed a raw value that never equalled Framing.COMPACT, so the compressor silently wrote a full container, and an unknown name was not rejected. __post_init__ now converts framing to the enum. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

- Compatibility: readers before v1.2 reject a \x89PTWS frame with "not a .ptwm container" (Python Decompressor) or "missing EOF sentinel (file may be truncated)" (Rust and _rust container functions); "bad magic" applies only to \x89PTWC. Same fix in the release notes. - CW2: add the terminal-ref rules (one byte only for node <= 14 and out <= 15, 0xF0-0xFE reserved, no needless escape), the compact role byte table and the 255 bound on edge indices. - Legacy containers keep the v1.1.0 Extension Table; explain why the v1.1 transcoder needs it and update the legacy size figures. - PTWC one-tensor cost is about 85-120 B for bf16 and 125-140 B for fp32; the PTWS default bound for smooth bf16 data is 49 B. - Reader limits: declared plane lengths no longer size allocations. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

Drop struct updates that set no field and Copy-type clones, so clippy --all-targets -D warnings passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

- encode_registry_for_table checks every registry op against the table before encoding and rejects tables beyond the u16 index space. A full imported table that lacked an op used to panic in the table builder. - set_compact_tables uses a checked u16 index. - Rans and Tans decode_bounded check the output length, which also covers the raw-fallback tag that carries no length prefix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

The overwrite prompt checked only the single-shard file. A multi-shard run replaces the directory without the last suffix, which was deleted with no prompt and no --force. A regular file in that place raised NotADirectoryError after the model was compressed. The guard now covers both destinations, and the replace step removes a file or a directory. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

claude added 30 commits October 27, 2026 17:33

claude added 2 commits October 28, 2026 05:11

style(tests): clear clippy lints in test code

a5a1548

Drop struct updates that set no field and Copy-type clones, so clippy --all-targets -D warnings passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H7NivSsfkoNxDZvZv1SnNm

khwstolle requested a review from khwstolle-bot October 28, 2026 14:45

khwstolle-bot removed their request for review October 28, 2026 14:46

khwstolle-bot reviewed Oct 28, 2026

View reviewed changes

Comment thread python/ptwm/cli/compress.py

khwstolle-bot reviewed Oct 28, 2026

View reviewed changes

Comment thread crates/ptwm-core/src/container.rs

khwstolle-bot reviewed Oct 28, 2026

View reviewed changes

Comment thread crates/ptwm-core/src/codecs/rans.rs

claude added 2 commits October 28, 2026 15:36

Navigation Menu

Repository navigation

Cut fixed per-container overhead with compact formats; fix cross-version decode and reader hardening - #6

Cut fixed per-container overhead with compact formats; fix cross-version decode and reader hardening#6
khwstolle wants to merge 34 commits into
masterfrom
claude/loving-keller-1ykxh8

khwstolle commented Oct 28, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

khwstolle commented Oct 28, 2026

Summary

Format changes

Existing defects fixed

Checklist

Notes for reviewers

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants