serialize.java

A bitpacking serialization library for Java. Part of the serialize family, wire compatible with the C++, C, Go, C#, Rust, JavaScript, Dart and Elixir libraries — the same values produce the same bytes in every implementation, so a stream written by one reads in any other. STANDARD.md — a verbatim vendored copy of the specification in mas-bandwidth/serialize, which CI checks for drift — is the authority on every byte.

Version 1.1.2 (SerializeUtil.VERSION).

Getting it

serialize.java ships as source today. The implementation is ready and the package is not yet published: there is no artifact on Maven Central under any coordinate, and there is no Maven or Gradle build here to produce one. Publishing it is a separate round.

Sixteen files under src/serialize/, zero dependencies, Java 17 language level. Take it either way:

Copy the package in — drop src/serialize/ into your own source tree and it compiles with the rest of it, no build changes at all.

Build a jar — the same javac line the Makefile uses, against any JDK 17 or newer:

git clone https://github.com/mas-bandwidth/serialize.java.git
javac --release 17 -d classes serialize.java/src/serialize/*.java
jar cf serialize.jar -C classes serialize

then compile and run against it with -cp serialize.jar. Write-side contracts are assert statements, so run with -ea while developing and without it in release — the two shapes the test suite covers.

Pin a release tag rather than tracking main. The newest is on the releases page: a release states a format version, and two endpoints interoperate only when they carry the same one.

The surface

One package, serialize, zero dependencies. The complete family operation set on three streams — WriteStream, ReadStream and MeasureStream — sharing the BitStream interface, so a single serialize function writes, reads and measures. Values travel in primitive-specialized holder cells (IntRef, LongRef, BoolRef, FloatRef, DoubleRef) that keep the hot path free of boxing, and in Ref<T> for object values (String, the 128-bit pair). USAGE.md teaches every operation by example.

• Raw bits: serializeBits (1–32), serializeBits64 (1–64), serializeAlign.
• Ranged integers: serializeInt, serializeInt64, serializeInt128 — offset from min in exactly the bit length of the range, unsigned-domain arithmetic so ranges wider than 2^63/2^127 are exact, zero bits for a degenerate min == max range on every width.
• Unsigned helpers and bool: serializeUint8 / 16 / 32 / 64, serializeUint128 (the UInt128Value pair), serializeBool.
• Floats: serializeFloat and serializeDouble, bit transparent both ways — every pattern legal, NaN payloads ride byte-for-byte; serializeCompressedFloat, quantizing in float32 with the standard's two roundings on each side.
• Bytes and strings: serializeBytes (aligned bulk copy, count agreed, not transmitted); serializeString (UTF-8 on the wire, payload validated on read in every mode); serializeWideString (one 32-bit group per UTF-16 code unit, no alignment anywhere).
• The relative integer: serializeIntRelative — the flag ladder for strictly increasing sequences over the domain 0 to 2^31 - 1, one bit for a difference of 1, every tier's reconstruction checked on read.
• Fixed point: serializeFixed at 8/16/32/64-bit storage and serializeFixed128 at 128-bit storage — Q formats, the raw scaled integer as an exact ranged offset, byte identical to serializeInt64 wherever storage fits 64 bits.
• 128-bit values: Int128Value / UInt128Value, two's complement pairs of long halves, mirroring the family's emulated pair types.
• Utilities: SerializeUtil — bitsRequired / 64 / 128, zig-zag conversion, the compressed-float step count.
• Composition: serializeObject, which runs a nested Serializer inline and contributes no bytes of its own — no framing, no length prefix, no alignment inserted around it.
• The bitpacker underneath: BitWriter and BitReader, the family wire in branchless 64-bit window loads. Every stream and both bitpackers expose reset(...) for allocation-free reuse.

Quick example

import serialize.*;

final class Player
{
    final IntRef health = new IntRef();
    final BoolRef alive = new BoolRef();
    final FloatRef heading = new FloatRef();
}

static boolean serializePlayer( BitStream stream, Player player )
{
    return stream.serializeInt( player.health, 0, 100 )
        && stream.serializeBool( player.alive )
        && stream.serializeFloat( player.heading );
}

byte[] buffer = new byte[64];                        // length a multiple of 8
WriteStream writer = new WriteStream( buffer, buffer.length );
serializePlayer( writer, player );                   // -> true
writer.flush();                                      // always flush before touching the bytes
int packetBytes = (int) writer.getBytesProcessed();  // 5 bytes: 7 + 1 + 32 bits

ReadStream reader = new ReadStream( buffer, packetBytes );
serializePlayer( reader, decoded );                  // -> true

Toolchain

The JDK is pinned per project, not taken from the system: tending/PINS.md records the exact version, download URL and SHA-256. dist/ is gitignored — re-fetch by the pinned URL, verify the hash, and unpack so the JDK sits at dist/jdk-21.0.12.1/Contents/Home. The library targets the Java 17 language level (javac --release 17), built and tested on the pinned JDK 21. A plain Makefile drives everything — no Maven, no Gradle:

make          # both shapes below
make test          # the suite with assertions on, the checked shape
make test-release  # the same suite with assertions off, the release shape

Testing

make test runs the suite with assertions enabled (-ea): writer contracts are assert statements, so this is the checked shape. make test-release runs the same suite with assertions disabled — the release shape, where asserts compile to nothing at runtime, matching the C++ library's serialize_assert under NDEBUG — which is what proves the read side's refusals are checks rather than asserts. Both are CI gates.

The suite runs every vector in conformance/, the family's shared corpus, vendored from mas-bandwidth/serialize and checked for drift by CI. The directory is discovered at run time rather than named in the source, and a vector whose operation the runner cannot drive fails rather than being skipped. An accepted vector must decode to the stated value and consume the stated bits; a vector marked writer = canonical is re-emitted through the write stream and compared byte for byte, flush included; a vector carrying measure_at_least is held to that floor on the measure stream. A refused vector must be refused, must leave the caller's scalar destination unwritten, and must leave the stream terminal, which is checked by behavior: every later step refuses too, and a further read fails, consumes no bits and writes nothing. Nothing regenerates its own expectations. The suite also pins the family's golden vectors byte for byte — the golden wire message covering every operation class, the discriminating compressed-float vectors (bit patterns, not tolerances), the string and wide-string pins, every relative-integer tier, and the fixed point shapes at every group count — plus a sabotage sweep proving every consumed bit of the golden stream is load bearing, refusal and terminality proofs for hostile input, and the measure bound.

interop/ takes it further: the CI interop job builds the C++ reference at a pinned release and runs it head to head with this port. Both halves write the same boundary message — every operation the standard defines, at its boundary values — and the files must be byte identical; each then decodes the other's bytes and re-encodes them exactly; both must refuse every truncation of the other's stream; and both run the corpus. The release candidate in this repository exchanges bytes with the reference on every push, so wire compatibility is measured rather than asserted. make interop MODE=write FILE=out.bin runs one exchange by hand.

Benchmarking for the serialize family lives in mas-bandwidth/schema's data-driven bench, which measures the generated codecs across every language on one corpus.

License

BSD 3-Clause, © Más Bandwidth LLC.