ZXC - Lossless Compression Built for Ultra-Fast Decode

ZXC is a fast lossless compression algorithm, targeting write-once, read-many workloads: data compressed once at build time, then decompressed on every device that reads it. It features an extremely fast decoder, with speeds of multiple GB/s per core: levels -1 to -6 decode 1.1x to 2.4x faster than LZ4 (lz4 --fast, lz4 or lz4hc, whichever matches the ratio) at an equal or better compression ratio. Level -7, the densest, compresses better than zstd -1 while decoding up to 2.8x faster.

ZXC also provides seekable archives for O(1) random access, in-place decompression, and dictionary compression for small data.

The ZXC format is fully specified in FORMAT.md and guarded by public conformance vectors. This repository is the reference implementation, provided as an open-source BSD 3-Clause licensed C library and a command line utility producing and decoding .zxc files, with official bindings for Rust, Python, Node.js, Go and WASM. The design is described in the whitepaper.

Benchmarks

Decompression speedup against the closest competitor at each ratio tier — Silesia corpus (202 MB), single thread, reproducible with lzbench and TurboBench:

Machine	-1 vs lz4 --fast	-3 vs lz4	-6 vs lz4hc -9	-7 vs zstd -1
Apple M2	2.41x	2.03x	1.52x	2.80x
Axion (Neoverse-V2)	1.93x	1.54x	1.29x	2.17x
EPYC 9B45 (Zen 5)	2.19x	1.65x	1.24x	2.49x
EPYC 7B13 (Zen 3)	1.82x	1.35x	1.15x	2.38x

ZXC also compresses smaller in each pairing: 61.76 % vs 62.15 %, 46.65 % vs 47.60 %, 36.28 % vs 36.75 % and 33.09 % vs 34.53 % of the original size.

Measured with lzbench 2.4.1 (from @inikep) built with MOREFLAGS="-march=native", on four reference machines: Apple M2 (Clang 21, macOS 26), Google Axion / Neoverse-V2 (GCC 14, GCP C4A), AMD EPYC 9B45 / Zen 5 (GCP C4D) and AMD EPYC 7B13 / Zen 3 (GCP C2D) — both x86 with SMT disabled. Re-run on every commit (latest logs); every number is reproducible with lzbench or TurboBench, where ZXC is merged alongside 70+ other codecs. Cycles per byte and memory figures live in the whitepaper.

Every level, on every CPU

All codecs

Apple M2 (ARM64)

Codec	Compression	Decompression	Ratio
zxc 0.15.0 -1	886 MB/s	13513 MB/s	61.76 %
zxc 0.15.0 -2	588 MB/s	11320 MB/s	53.86 %
zxc 0.15.0 -3	267 MB/s	9707 MB/s	46.65 %
zxc 0.15.0 -4	194 MB/s	8790 MB/s	43.43 %
zxc 0.15.0 -5	114 MB/s	8118 MB/s	40.83 %
zxc 0.15.0 -6	14.5 MB/s	6884 MB/s	36.28 %
zxc 0.15.0 -7	9.71 MB/s	5054 MB/s	33.09 %
lz4 1.10.0 --fast -17	1347 MB/s	5609 MB/s	62.15 %
lz4 1.10.0	814 MB/s	4774 MB/s	47.60 %
lz4hc 1.10.0 -9	48.4 MB/s	4522 MB/s	36.75 %
lzav 5.17 -1	686 MB/s	3876 MB/s	39.91 %
snappy 1.3.1	661 MB/s	3258 MB/s	47.85 %
zstd 1.5.7 --fast --1	723 MB/s	2538 MB/s	41.01 %
zstd 1.5.7 -1	645 MB/s	1803 MB/s	34.53 %
zstd 1.5.7 -3	375 MB/s	1680 MB/s	31.20 %
zlib 1.3.2 -1	135 MB/s	378 MB/s	36.45 %

Google Axion (Neoverse-V2, ARM64)

Codec	Compression	Decompression	Ratio
zxc 0.15.0 -1	885 MB/s	9564 MB/s	61.76 %
zxc 0.15.0 -2	593 MB/s	7895 MB/s	53.86 %
zxc 0.15.0 -3	256 MB/s	6571 MB/s	46.65 %
zxc 0.15.0 -4	186 MB/s	6105 MB/s	43.43 %
zxc 0.15.0 -5	109 MB/s	5683 MB/s	40.83 %
zxc 0.15.0 -6	13.5 MB/s	4950 MB/s	36.28 %
zxc 0.15.0 -7	8.82 MB/s	3563 MB/s	33.09 %
lz4 1.10.0 --fast -17	1278 MB/s	4954 MB/s	62.15 %
lz4 1.10.0	730 MB/s	4258 MB/s	47.60 %
lz4hc 1.10.0 -9	43.2 MB/s	3850 MB/s	36.75 %
lzav 5.17 -1	576 MB/s	2949 MB/s	39.91 %
snappy 1.3.1	566 MB/s	2295 MB/s	47.85 %
zstd 1.5.7 --fast --1	302 MB/s	2291 MB/s	41.01 %
zstd 1.5.7 -1	523 MB/s	1642 MB/s	34.53 %
zstd 1.5.7 -3	329 MB/s	1525 MB/s	31.20 %
zlib 1.3.2 -1	115 MB/s	389 MB/s	36.45 %

AMD EPYC 9B45 (Zen 5, x86_64)

Codec	Compression	Decompression	Ratio
zxc 0.15.0 -1	868 MB/s	11609 MB/s	61.76 %
zxc 0.15.0 -2	585 MB/s	10332 MB/s	53.86 %
zxc 0.15.0 -3	253 MB/s	8335 MB/s	46.65 %
zxc 0.15.0 -4	183 MB/s	7508 MB/s	43.43 %
zxc 0.15.0 -5	110 MB/s	7014 MB/s	40.83 %
zxc 0.15.0 -6	15.3 MB/s	6045 MB/s	36.28 %
zxc 0.15.0 -7	10.2 MB/s	4724 MB/s	33.09 %
lz4 1.10.0 --fast -17	1284 MB/s	5305 MB/s	62.15 %
lz4 1.10.0	766 MB/s	5052 MB/s	47.60 %
lz4hc 1.10.0 -9	45.4 MB/s	4861 MB/s	36.75 %
lzav 5.17 -1	699 MB/s	3603 MB/s	39.91 %
snappy 1.3.1	564 MB/s	2420 MB/s	47.89 %
zstd 1.5.7 --fast --1	664 MB/s	2451 MB/s	41.01 %
zstd 1.5.7 -1	606 MB/s	1898 MB/s	34.53 %
zstd 1.5.7 -3	369 MB/s	1733 MB/s	31.20 %
zlib 1.3.2 -1	134 MB/s	392 MB/s	36.45 %

AMD EPYC 7B13 (Zen 3, x86_64)

Codec	Compression	Decompression	Ratio
zxc 0.15.0 -1	724 MB/s	8155 MB/s	61.76 %
zxc 0.15.0 -2	481 MB/s	6807 MB/s	53.86 %
zxc 0.15.0 -3	214 MB/s	5252 MB/s	46.65 %
zxc 0.15.0 -4	157 MB/s	4952 MB/s	43.43 %
zxc 0.15.0 -5	93.3 MB/s	4779 MB/s	40.83 %
zxc 0.15.0 -6	11.9 MB/s	4290 MB/s	36.28 %
zxc 0.15.0 -7	7.97 MB/s	3188 MB/s	33.09 %
lz4 1.10.0 --fast -17	1108 MB/s	4485 MB/s	62.15 %
lz4 1.10.0	638 MB/s	3883 MB/s	47.60 %
lz4hc 1.10.0 -9	37.1 MB/s	3731 MB/s	36.75 %
lzav 5.17 -1	459 MB/s	2955 MB/s	39.91 %
snappy 1.3.1	507 MB/s	1837 MB/s	47.89 %
zstd 1.5.7 --fast --1	486 MB/s	1784 MB/s	41.01 %
zstd 1.5.7 -1	444 MB/s	1339 MB/s	34.53 %
zstd 1.5.7 -3	232 MB/s	1178 MB/s	31.20 %
zlib 1.3.2 -1	106 MB/s	358 MB/s	36.45 %

Effective throughput

What is Effective Throughput?

Raw decode speed misses half the picture: in real workloads (asset streaming, container pulls, microservice payloads), the decoder is fed by a compressed-byte source - disk, network, inter-core - whose bandwidth is the bottleneck. The right question is how much original data is delivered per MB of compressed input.

Formula: Effective (MB/s) = Decode × 100 / Ratio (%): combines decode speed and ratio in one number. Every ZXC level from -1 to -7 sits above LZ4 on every architecture, peaking at 2.18x on Apple Silicon and ranging 1.38x–1.81x on x86 and ARM cloud platforms for levels -1 to -6. The density-optimized ULTRA level -7 clears LZ4 as well (1.18x–1.52x), at a 33.09% ratio.

Features

• 1.1x to 2.4x faster decode than LZ4 at levels -1 to -6, at an equal or better ratio. Level -7 trades that lead for density: it decodes 2.2x to 2.8x faster than zstd -1, at a better ratio (benchmarks).
• Write once, read many. The encoder does the heavy lifting, so every device that reads the data decodes faster: content delivery, game assets, app bundles, firmware. Gains are largest on modern CPUs.
• O(1) random access. An optional seek table (-S) decompresses any block without reading the rest.
• Decodes in place. One buffer instead of two, zero allocations with a static context: made for firmware, FOTA and bootloaders (details).
• Small payloads too. A trained dictionary recovers ratio on 4–128 KB blocks (details).
• Runs everywhere. x86_64, ARM64, ARMv7, ARMv6, RISC-V, POWER, s390x, i386, with hand-tuned SIMD (AVX2/AVX-512/NEON).
• Production-grade. Continuously fuzzed by OSS-Fuzz, ASan/UBSan/Valgrind-clean, a specified wire format with conformance vectors, signed releases, BSD-3-Clause.

Used in ClickHouse (experimental column codec) · Packaged in Debian 14, Ubuntu 26.10, Homebrew, vcpkg, Conan, Winget · Benchmarked in lzbench & TurboBench

Compression Levels

• Level 1, 2 (Fast): Optimized for real-time assets (Gaming, UI).
• Level 3, 4 (Balanced): A strong middle-ground offering efficient compression speed and a ratio superior to LZ4.
• Level 5 (Compact): A good choice for Embedded and Firmware. Better compression than LZ4 and significantly faster decoding than Zstandard.
• Level 6 (Density): Beats LZ4HC on both axes — better ratio and faster decode on every measured platform — while staying in the multi-GB/s decode class. Best for Archival and write-once / read-many workloads where compression time is amortized over many reads.
• Level 7 (Ultra): Maximum density. Deep parse plus Huffman-coded literals and tokens (11-bit codes) push the ratio past zstd -1 while decoding more than twice as fast. Choose it when storage or bandwidth dominates but decode must remain fast; compression is the slowest tier.

Usage

1. CLI

The CLI is perfect for benchmarking or manually compressing assets.

# Compress. -z is implied, and the output name defaults to <input>.zxc
zxc assets.tar                        # level 3 (default) -> assets.tar.zxc
zxc -z -5 assets.tar assets.tar.zxc   # level 5
zxc -z -S assets.tar assets.tar.zxc   # seekable: O(1) random-access decompression

# Decompress. "unzxc" (POSIX installs) is an alias for "zxc -d"
zxc -d assets.tar.zxc assets.tar
unzxc assets.tar.zxc assets.tar

# Decompress to stdout. "zxccat" (POSIX installs) runs "zxc -dcf" on each file
zxccat logs-1.zxc logs-2.zxc | grep ERROR

# Read a compressed file page by page in less (POSIX installs)
zxcless app.log.zxc

# Search compressed files with grep's options (POSIX installs)
zxcgrep -n ERROR logs-*.zxc

# Benchmark mode (testing speed on your machine)
zxc -b assets.tar

Every option is in zxc --help and the man page.

Using with tar

ZXC works as a drop-in external compressor for tar (reads stdin, writes stdout, returns 0 on success):

# GNU tar (Linux)
tar -I 'zxc -5' -cf archive.tar.zxc data/
tar -I 'zxc -d' -xf archive.tar.zxc

# bsdtar (macOS)
tar --use-compress-program='zxc -5' -cf archive.tar.zxc data/
tar --use-compress-program='zxc -d' -xf archive.tar.zxc

# Pipes (universal)
tar cf - data/ | zxc > archive.tar.zxc
zxc -d < archive.tar.zxc | tar xf -

2. API

ZXC provides a thread-safe API with two usage patterns. Parameters are passed through dedicated options structs, making call sites self-documenting and forward-compatible. Buffers are caller-allocated with explicit bounds, calls are stateless, checksum validation is optional, block sizes run from 4 KB to 2 MB (powers of two), and streaming is multi-threaded with auto-detection of the CPU core count.

#include "zxc.h"

// Compression
uint64_t bound = zxc_compress_bound(src_size);
zxc_compress_opts_t c_opts = {
    .level            = ZXC_LEVEL_DEFAULT,
    .checksum_enabled = 1,
    /* .block_size = 0 -> 512 KB default */
};
int64_t compressed_size = zxc_compress(src, src_size, dst, bound, &c_opts);

// Decompression
zxc_decompress_opts_t d_opts = { .checksum_enabled = 1 };
int64_t decompressed_size = zxc_decompress(src, src_size, dst, dst_capacity, &d_opts);

The same options structs drive the other entry points: zxc_stream_compress() / zxc_stream_decompress() for multi-threaded file streaming, reusable zxc_cctx / zxc_dctx contexts for tight loops where per-call malloc/free overhead matters (settings are sticky, so passing NULL reuses those given at creation), and .seekable = 1 to append a seek table for O(1) random-access decompression.

👉 See complete examples and advanced usage — stream API, reusable contexts, seekable readers, dictionaries and numeric pre-filters, as full compilable programs.

Installation

ZXC is packaged across major ecosystems and kept current by their maintainers:

Ecosystem	Install
vcpkg	vcpkg install zxc, or "dependencies": ["zxc"] in vcpkg.json
Conan	conan install -r conancenter --requires="zxc/[*]" --build=missing, or zxc/[*] under [requires] in conanfile.txt
Homebrew	brew install zxc
winget (Windows 10 1709+)	winget install hellobertrand.zxc
Rust / Python / Node.js	cargo add zxc-compress · pip install zxc-compress · npm install zxc-compress

The vcpkg, Conan Center and winget recipes are maintained by their respective communities, so they can lag a release behind; if one does, open an issue on that registry's index repository.

From a release archive

Prebuilt binaries for Linux and Windows (x86_64 and ARM64) and macOS (ARM64) are on the Releases page, with a signed checksum manifest and build attestations: verification steps.

In your project

find_package(zxc REQUIRED)          # find_package(zxc CONFIG REQUIRED) via vcpkg or Conan
target_link_libraries(myapp PRIVATE zxc::zxc_lib)

cc myapp.c $(pkg-config --cflags --libs libzxc) -o myapp

Vendoring zxc instead — CMake FetchContent or add_subdirectory(), a Meson subproject or WrapDB — and building from source, with the full option table and the PGO workflow: docs/INSTALL.md.

Packaging Status

Language Bindings

Official wrappers maintained in this repository:

Language	Package Manager	Install Command	Documentation	Author
Rust	crates.io	cargo add zxc-compress	README	@hellobertrand
Python	PyPI	pip install zxc-compress	README	@nuberchardzer1
Node.js	npm	npm install zxc-compress	README	@hellobertrand
Go	go get	go get github.com/hellobertrand/zxc/wrappers/go	README	@hellobertrand
WASM	npm	npm install zxc-wasm	README	@hellobertrand

Community-maintained bindings:

Language	Package Manager	Install Command	Repository	Author
Go	pkg.go.dev	go get github.com/meysam81/go-zxc	https://github.com/meysam81/go-zxc	@meysam81
Nim	nimble	nimble install zxc	https://github.com/openpeeps/zxc-nim	@georgelemon
Free Pascal	Build from source	Clone the repository	https://github.com/Xelitan/Free-Pascal-port-of-ZXC-compressor-decompressor	@Xelitan

In-Place Decompression

A normal decode needs two buffers: the archive and the output, so peak memory is compressed + decompressed. ZXC can decode inside a single buffer: you place the archive at its end, and the decoder writes the output from its start.

buffer:  [ output → → →          | archive → → → ]
           ^ write cursor           ^ read cursor

Both cursors move right, and a ZXC block never expands, so with a small safety margin the write cursor provably never catches the read cursor. Peak memory drops to roughly decompressed alone. This is the shape of firmware images, FOTA payloads, bootloaders, game assets: anything read-only that already sits in RAM.

One buffer:

size_t need = zxc_decompress_inplace_bound(archive, archive_size);   // reads header + footer only
uint8_t* buf = malloc(need);
memcpy(buf + (need - archive_size), archive, archive_size);          // archive at the end
int64_t n = zxc_decompress_inplace(buf, need, archive_size, NULL);   // decode from buf[0]
// buf[0 .. n) now holds the decompressed data

One buffer, zero allocations. zxc_decompress_inplace still allocates its decode context for the duration of the call. Where the library must not touch an allocator at all (kernel code, heapless targets, fixed memory budgets), hand it a context living in memory you own:

// Once, up front: sized for the block size your archives use.
size_t ws_size = zxc_static_dctx_workspace_size(64 * 1024);
void* ws = aligned_alloc(64, ws_size);                                // or .bss, kmalloc, vmalloc
zxc_dctx* dctx = zxc_init_static_dctx(ws, ws_size, 64 * 1024);

// Per archive: no allocation inside the library.
int64_t n = zxc_decompress_inplace_dctx(dctx, buf, need, archive_size, NULL);

A static context accepts only archives compressed with its block size (-B), and no dictionary; anything else is refused with an error before decoding.

What it costs. The margin is one block, plus 8–12 bytes per block of framing, plus ~2 KB of tail; the static context is three to four blocks. So the block size decides both, and small payloads want small blocks:

Block size (-B)	Margin, 8 MB payload	Static context
4 KB	22–30 KB (0.3–0.4 %)	16 KB
64 KB	67–68 KB (0.8 %)	208 KB
512 KB (default)	514 KB (6.3 %)	1.6 MB

The higher margin is with block checksums, which the CLI writes by default. Smaller blocks cost some ratio and decode speed; measure on your data. Always size the buffer with zxc_decompress_inplace_bound rather than the formula.

What it guarantees. An undersized buffer is rejected with ZXC_ERROR_DST_TOO_SMALL. An archive whose output would reach input not yet read (padding, forged block sizes) is rejected with ZXC_ERROR_CORRUPT_DATA. Never silent corruption: both entry points are covered by a dedicated fuzzer.

Dictionary Compression

For workloads compressed in small blocks (4 KB–128 KB), a pre-trained dictionary dramatically improves compression ratio. It prefills the LZ77 sliding window at the start of each block, so the benefit is per-block: the smaller the block, the less history of its own it has and the more it leans on the dictionary. That holds for a single small payload as much as for a large one split into many small blocks — anywhere early bytes would otherwise have nothing to match against.

A .zxd also carries a shared literal Huffman table, trained on the post-LZ literal distribution of the corpus: blocks it encodes well drop their own 128-byte table header, a fixed cost small blocks cannot amortize. It codes literals at levels 6-7 only, and the CLI handles it end to end — --train always writes one, -D always loads it.

Typical use cases: JSON API responses, small game assets, structured logs, key-value store records, RPC messages, and any large but homogeneous corpus compressed in small blocks for random access (e.g. seekable archives).

# Train from a corpus of similar files. Without -o: ./dictionary_<dict_id>.zxd
zxc --train samples/*.json
zxc --train -o corpus.zxd samples/*.json     # -o also accepts a directory

# The same dictionary is required to decompress: pass it with -D, there is no auto-lookup
zxc -z -D corpus.zxd input.json
zxc -d -D corpus.zxd input.json.zxc

The dictionary is an external .zxd file — content plus shared literal table — referenced by a 32-bit dict_id in the archive header that covers both parts. Decompressing an archive that needs one without supplying it returns ZXC_ERROR_DICT_REQUIRED; supplying the wrong one returns ZXC_ERROR_DICT_MISMATCH. Training and attaching a dictionary from C: EXAMPLES.md and API.md §11b. Wire format: FORMAT.md §12.

Block Size Tuning

The default block size is 512 KB, tuned for bulk/archival workloads where ratio and decompression throughput matter most. For memory-constrained or streaming use cases, 256 KB blocks halve the decompression context and cut the compression context by ~40 % at a small cost in ratio and decompression speed.

Why larger blocks help: Each block starts with a cold hash table, so the LZ match-finder has no history and produces more literals until the table warms up. Doubling the block size halves the number of cold-start penalties, improving both ratio and decompression speed.

Block Size	cctx memory	dctx memory	Ratio (level -3)	Decompression gain vs 256 KB
256 KB	~1.04 MB	~514 KB	47.21%	—
512 KB (default)	~1.79 MB	~1.0 MB	46.65% (−0.56 pp)	+1% to +8% depending on CPU

# CLI — fall back to 256 KB blocks (e.g. embedded / streaming)
zxc -B 256K -5 input_file output_file

# API
zxc_compress_opts_t opts = {
    .level      = ZXC_LEVEL_COMPACT,
    .block_size = 256 * 1024,
};

Guideline: Stick with 512 KB (default) for bulk compression pipelines, CI/CD asset packaging, and high-throughput servers. Use 256 KB (-B 256K) for streaming, embedded, or memory-constrained environments.

Format & Conformance

The ZXC on-disk wire format is fully specified in docs/FORMAT.md (format version 9), so any third party can build an independent, interoperable decoder.

Upgrading? The current format is v9 — block checksums now cover the decoded bytes and are seeded by block position, the footer carries an optional archive digest, and the seek table became self-validating groups announced by a header flag. Like every break before it, this one is clean: v9 tools reject v8 archives (see docs/MIGRATION.md to convert).

Two complementary, byte-frozen suites guard that format:

• Decoder conformance — conformance/ ships public reference vectors, frozen per format version: valid/*.zxc streams paired with their expected decompressed output, plus invalid/*.zxc streams that a correct decoder must reject, at least one per row of the format's error table. Point your own decoder at them to prove interoperability — no dependency on this implementation. Run locally via the conformance CTest.
• Wire-format stability — tests/format/ pins the exact bytes the encoder emits for every block type and integrity field. A CI job (vector-stability.yml) fails on any single-byte drift, so an encoder change can only ever be deliberate.

The distinction: conformance freezes decoder behaviour (decode(x) == expected), while the golden suite freezes the encoder's bytes. Together they make the format both interoperable and stable.

Safety & Quality

• Unit Tests: Comprehensive test suite with CTest integration.
• Continuous Fuzzing: Enrolled in Google OSS-Fuzz, which fuzzes eight harnesses (roundtrip, decompress, block, in-place, push streaming, multi-threaded FILE* streaming, seekable, dictionary) around the clock, covering over 80 % of the library's lines (coverage history). The same harnesses run under ClusterFuzzLite (ASan + UBSan) on every pull request touching the library.
• Static Analysis: Checked with Cppcheck & Clang Static Analyzer.
• CodeQL Analysis: GitHub Advanced Security scanning for vulnerabilities.
• Snyk: Continuous security and code analysis for dependencies and source.
• Code Coverage: Automated tracking with Codecov integration.
• Dynamic Analysis: Validated with Valgrind and ASan/UBSan in CI pipelines.
• Safe API: Explicit buffer capacity is required for all operations.

License & Credits

ZXC Copyright © Bertrand Lebonnois and contributors. Licensed under the BSD 3-Clause License. See LICENSE for details.

Third-Party Components:

• rapidhash by Nicolas De Carli (MIT) - Used for high-speed, platform-independent checksums.

Acknowledgements:

• PivCo-Huffman (PIVoted COding) by Marcin Żukowski - the level-ordered layout and merge-based decode of ZXC's Huffman sections follow its design; implemented independently here. Special thanks to Dougall Johnson, whose idea of jointly nudging flat and length codes is behind the level 6 / 7 decode speedups.