[timtofan]

Why

Backend gzip inflation consumes about 20–30% of CPU in live profiles and about 43% on a large wildcard request. Faster decoding reduces request cost and tail latency without changing metric evaluation.

Architecture

The existing streaming backend reader uses zlib-ng 2.3.3 through its native zng_* API. The default Make build downloads a commit-pinned, SHA-256-verified source archive, runs the library's tests, and links a private static archive. Runtime CPU dispatch remains enabled.

This extends the fork's existing native decoder; system zlib lacks the measured decode performance. Separate symbols keep Cairo/libpng on system zlib and leave response compression unchanged. There is no preload shim or system-library replacement. The fork-specific integration can be removed if the common backend reader adopts an equivalent implementation.

Before / after

Before	After
Default native build inflates with system zlib	Default native build links the pinned zlib-ng decoder
Cairo/libpng and response compression use their current implementations	Same implementations and output bytes
Portable builds use Go gzip	Same fallback; system-zlib native build also remains available

Usage

make build activates the dependency in the existing native build. The existing CARBONAPI_STREAM_RENDER=1 and CARBONAPI_NATIVE_GZIP=1 flags select this reader. No new runtime knob is introduced.

make build TAGS='-tags cairo,nativegzip' builds the system-zlib alternative; make nocairo uses the portable Go path. The README documents prerequisites, checksum pinning and cache cleanup. Docker and make test exercise the default integration; the latter also checks both decoder fallbacks.

Validation

• Complete race suite passed on macOS/arm64 and production Linux/amd64 with Cairo; default make test, both fallback tests and golangci-lint 2.14.0 passed locally.
• Extended differential tests cover compression levels, incompressible input, optional gzip headers and fragmented reads. Existing tests cover reset, concatenated members, corruption, truncation and streaming protobuf parity.
• Production-host replay of this build: 144 responses across 24 queries. All matched: 22 queries byte-for-byte (including PNG), two pickle queries by decoded structure and exact float64 bits because both builds vary pickle key order. Aggregate measured CPU was 6.38→4.44s (~30% lower); the large query's median was 679→488ms (~28% lower). Allocated bytes were essentially unchanged.
• Earlier single-host live A/B/A on the previous deployed build, using the same library through an inflate-only diagnostic shim: exact-log p99 593→519→619ms; CPU per render normalized against five controls was approximately 9–15% lower.

The earlier live trial is evidence for a partial improvement, not proof that this PR fixes the six-VM regression. Rare worst-case latency and sustained fleet peak load remain unvalidated. GitHub CI passed on oldstable, stable and tip; CodeQL also passed.

Rollback / risk

Restore the previous binary or build without zlibng. Existing startup flags can select Go decoding. First native builds now require downloading and compiling the pinned dependency; subsequent builds reuse .build/zlib-ng. No memory-reduction claim is made.