Re-vendor the expanded corpus: 349 vectors through the reader, the writer and the measure

[gafferongames]

STANDARD.md and conformance/ come from mas-bandwidth/serialize@7e0515e, byte for byte. 18 files, 349 vectors.

What the runner gained

ConformanceTests drove two operations, int_relative and int128, because that was all the vendored corpus held. It is now the runner STANDARD.md specifies.

• Discovery, not a list. conformance/*.txt is enumerated at run time. An empty directory fails the run.
• No skipping. A vector whose operation the runner cannot drive fails, and so does a vector naming a parameter the operation does not take. operationTakesParam mirrors the reference.
• One step machine over the single operation files and the sequence, object and message files. A single operation vector is a one or two step sequence built from the record's own parameters, preceding_bits becoming a leading bits step, so the sequence files cannot drift away from the operation files.
• Patterns, never floats. Every integer width and the float, double and compressed_float bit patterns compare as 128 bit two's complement patterns, so a hexadecimal expectation and its decimal twin are one expectation, NaN compares unequal to itself, and a quieted signaling bit is visible. Java has no unsigned integers and no 128 bit type, so the vector numbers travel as BigInteger and land in the operation's own width at the call site.
• Terminality by behavior. Every refused vector now checks that every later step of the sequence refuses too, and that a further read the vector does not name fails, consumes no bits and writes nothing to its own destination.
• The writer leg. A vector marked writer = canonical is re-emitted through the write stream and the whole emitted stream is compared byte for byte, flush included, which is where the trailing bits obligation bites.
• The measure leg. A vector carrying measure_at_least runs through the measure stream as a floor.
• The buffer contract. Every stream is presented with the eight bytes of slack the reader loads through, filled with 0xA5, the empty stream of a zero bit read included.
• Lexical rules. # begins a comment at the start of a line and nowhere else. Numbers are signed decimal or 0x hexadecimal, parsed to 128 bits.

349 vectors from 18 file(s): 195 writer checks, 9 measure checks

Identical counts to the reference runner, in both make test (assertions on) and make test-release (assertions off). No corpus vector fires an assertion under -ea: a refusal is a refusal in both shapes.

What the rulings implied for the library

object is a required operation and Java had no spelling for it. STANDARD.md makes serialize_object composition rather than an encoding, and names it among the operations that must refuse on a stream that has already failed. BitStream.serializeObject( Serializer ) runs a nested object inline, contributes no bytes of its own, and on a read stream consults the failure latch before it invokes the object. object.txt's twins are what hold it: the same operations nested and flat must produce the same bytes and the same consumption.

The bits width bound now runs at the caller's width. This is the narrowing class of serialize 1.16.1, and it is present in this port. STANDARD.md: the bound value < 2^bits holds at every width in [1,64]. serializeBits64 narrows a 64 bit value to a 32 bit group at widths of 32 or fewer:

assert checkBits64( bits );                 // width only
if ( bits <= 32 )
{
    writer.writeBits( (int) value.value, bits );   // the bound applies here, to the narrowed group
}

The only bound left was the one BitWriter.checkWriteBits applies to the already narrowed group, so a value of 2^32 + 5 in a four bit field arrived as 5 and passed. Above 32 bits the high group's own check happens to see the top bits, so the defect is exactly the widths at or below 32. SerializeUtil.valueFitsInBits( long value, int bits ) is now the one place the rule lives, called by both WriteStream.serializeBits64 and MeasureStream.serializeBits64 before any narrowing, mirroring serialize::value_fits_in_bits. The measure stream had no value bound at all.

What else was grepped and found clean. serializeIntRelative takes int, which is the operation's own API type, so a wider caller value narrows visibly at the call site rather than inside the library. serializeString and serializeWideString compare utf8.length and String.length() against bufferSize, all int already, so there is no narrowing between the length and its check. serializeFixed asserts Long.compareUnsigned( offset, rawRange ) on the 64 bit offset before writing the 32 bit groups. serializeCompressedFloat narrows maxIntegerValue to int only to hand it to bitsRequired, which reads it unsigned, and the value is bounded below 2^32 by the normative clamp.

Rulings already satisfied, confirmed against the corpus. No trap or abort on malformed input, Java arithmetic wraps. Every read consults the failure state first, degenerate ranged reads, a zero count bytes, an already aligned align and a degenerate fixed included. The offset form of the range check at every width, Integer.compareUnsigned, Long.compareUnsigned and UInt128Value.compareUnsigned. Invalid UTF-8 as Unicode Table 3-7. wstring groups above 0xFFFF, unpaired surrogates and zero groups refused, pairs accepted, no alignment anywhere. compressed_float in float32 with two roundings and the normative integer clamp. The measure charging 7 bits per alignment performing operation.

Negative controls

The corpus is an instrument, and an instrument nobody has seen go red is a claim rather than a measurement. Each sabotage was reverted.

1. Unknown operation

conformance-controls/unknown-operation.txt copied into conformance/, run, then deleted. It is not committed.

$ make conformance
    conformance unknown-operation.txt: control-a-runner-must-fail-on-an-operation-it-cannot-drive
        FAILED: no runner for parameter 'whatever' on operation 'no_such_operation' [unknown-operation.txt]
    350 vectors from 19 file(s): 195 writer checks, 9 measure checks
370 tests, 2020 checks, 1 failed
TESTS FAILED
make exit: 2

2. The alignment padding zero check deleted

BitReader.readAlign reduced to consuming the padding and always returning true. Ten vectors across five files go red.

    conformance align.txt: align-refuse-one-non-zero-pad-bit
        FAILED: the read succeeded, the corpus requires refusal [align.txt]
    conformance align.txt: align-refuse-four-non-zero-pad-bits
        FAILED: the read succeeded, the corpus requires refusal [align.txt]
    conformance align.txt: align-refuse-seven-non-zero-pad-bits
        FAILED: the read succeeded, the corpus requires refusal [align.txt]
    conformance align.txt: align-refuse-only-the-highest-pad-bit-set
        FAILED: the read succeeded, the corpus requires refusal [align.txt]
    conformance align.txt: align-refuse-only-the-lowest-pad-bit-set
        FAILED: the read succeeded, the corpus requires refusal [align.txt]
    conformance bytes.txt: bytes-zero-count-refuse-non-zero-align-padding
        FAILED: the read succeeded, the corpus requires refusal [bytes.txt]
    conformance bytes.txt: bytes-refuse-non-zero-align-padding
        FAILED: the read succeeded, the corpus requires refusal [bytes.txt]
    conformance object.txt: object-refusal-propagates-out-of-the-nesting
        FAILED: the read succeeded, the corpus requires refusal [object.txt]
    conformance sequence.txt: sequence-refusal-is-terminal-after-bad-alignment
        FAILED: the read succeeded, the corpus requires refusal [sequence.txt]
    conformance string.txt: string-refuse-non-zero-align-padding
        FAILED: the read succeeded, the corpus requires refusal [string.txt]
368 tests, 2025 checks, 10 failed
TESTS FAILED

3. min dropped from the ranged int reconstruction

ReadStream.serializeInt's value.value = unsignedValue + min reduced to value.value = unsignedValue. Six vectors go red, exactly the ones with a non-zero min. The degenerate range vectors take the early return and the [0,N] ranges are unaffected, which is the right shape.

    conformance int.txt: int-negative-min-golden-field
        FAILED: step 1 decoded 0x0000000000000000000000000000003F, the corpus states -37 [int.txt]
    conformance int.txt: int-negative-min-accepts-its-minimum
        FAILED: step 1 decoded 0x00000000000000000000000000000000, the corpus states -100 [int.txt]
    conformance int.txt: int-negative-min-accepts-its-maximum
        FAILED: step 1 decoded 0x000000000000000000000000000000C8, the corpus states 100 [int.txt]
    conformance int.txt: int-full-int32-range-golden-field
        FAILED: step 1 decoded 0x00000000000000000000000078A432EB, the corpus states -123456789 [int.txt]
    conformance int.txt: int-full-int32-range-minimum
        FAILED: step 1 decoded 0x00000000000000000000000000000000, the corpus states -2147483648 [int.txt]
    conformance int.txt: int-full-int32-range-maximum
        FAILED: step 1 decoded 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF, the corpus states 2147483647 [int.txt]
368 tests, 2021 checks, 6 failed
TESTS FAILED

4. The narrowing fix sabotaged back

checkBits64 returned to the pre-fix form, taking only the width, so the sole remaining bound is the one applied to the already narrowed group.

$ make test
    write contract: a bits64 value is bounded at the caller's width, before it narrows
        FAILED: the write stream must reject a value that does not fit four bits
458 tests, 4192 checks, 1 failed
TESTS FAILED
make exit: 2

Reverted, the same run is 458 tests, 4196 checks, 0 failed.

CI

SERIALIZE_TAG is untouched at v1.16.0. The interop job now builds the reference conformance runner from a second pin, SERIALIZE_CORPUS_COMMIT, the commit this repository vendors STANDARD.md and conformance/ from. The two pins answer different questions: the wire compatibility pin is a released tag and must stay one, and a runner older than the corpus cannot drive the operations the corpus carries, so the runner travels with the vectors.

Version

SerializeUtil.VERSION and the README line move to 1.1.2. The README line naming the current published release tag is untouched. No tag.

🤖 Generated with Claude Code