[timtofan]

What issue is this change attempting to solve?

Why

Long-range sparse renders spend CPU on point-by-point replica merging and allocate a temporary absence array per decoded metric. Most of that work is avoidable when packed blocks contain only positive-zero values. Reducing this overhead leaves more CPU and allocation headroom for concurrent requests.

Architecture

The existing flow remains backend response → streamed protobuf decoder → packed metrics → replica merge → dense metrics → expression evaluation.

The decoder reuses absence scratch storage within one response after packing has copied its flags into an owned bitmap. It detaches scratch storage when a dense metric retains the array and discards capacity above 1 MiB. No scratch storage is shared between requests. Reads overwrite every byte before bool normalization, so reused arrays do not need an extra clearing pass.

The merger combines absence bitmaps for 64-point blocks only when every compatible replica has no nonzero value bits. All values in such a block are positive zero, so only absence healing can change the result. Dense blocks, nonzero values, negative zero and NaNs use the existing point-wise path. Replica precedence, schema compatibility, mismatch accounting and the dense output contract are preserved. This extends the existing packed implementation without adding a new configuration path.

Before / after

Before	After
A temporary absence array is allocated for each sparse metric and discarded after packing	A bounded array is reused within the response after ownership transfers to the packed bitmap
Every point is visited during replica merging	Eligible zero blocks combine absence masks; other blocks keep the point-wise merge
Downstream functions receive dense point arrays	Same dense arrays and bit-level values

Usage

The current default packed backend render path uses both changes automatically. Existing dense decoding and CARBONAPI_PACK_BACKEND_POINTS=0 remain available. No new runtime switch or deployment override is introduced.

How does this change solve the problem? Why is this the best approach?

Both changes use the existing representation and preserve the expression-evaluation boundary. They target allocation churn and repeated zero-point inspection without changing query semantics, response ordering, concurrency, or scheduling. Removing the final dense materialization would require a broader evaluator change and is outside this PR.

How can we be sure this works as expected?

Correctness

• New mixed packed/dense metric test covers scratch ownership, shorter successors, buffer growth, empty repeated fields and fragmented reads against the reference decoder.
• New zero-block merge oracle compares exact floating-point bits and statistics with the dense merge across all three replica modes, exact/approximate comparison, partial blocks, schema differences, mixed representations, NaNs, infinities and negative zero.
• Existing packed merge, decoder differential, truncation and wire-variant tests remain in the validation suite.

Benchmarks

Apple M4 Max, Go 1.26.6, GOMAXPROCS=2. Frozen test binaries from master 6be69a2 and this change were run in A/B/B/A order; each table entry is the median of ten runs. These are isolated synthetic fixtures, not a whole-service latency measurement.

Benchmark	Before	After
Sparse decoder, allocated bytes / 100 metrics × 10,000 points	1,599,989 B	586,162 B (−63%)
Zero decoder, allocated bytes / 100 metrics × 10,000 points	1,549,540 B	535,749 B (−65%)
Sparse decoder time	1.526 ms	1.397 ms (−8%)
Zero decoder time	1.356 ms	1.224 ms (−10%)
Sparse replica merge, normal mode / 3 replicas × 10,080 points	27.664 µs	10.286 µs (−63%)
Sparse replica merge, check mode	66.913 µs	12.869 µs (−81%)
Zero replica merge, check mode	68.081 µs	8.364 µs (−88%)

Dense controls were within approximately 3%, with unchanged allocation counts. Merge output allocations are unchanged because downstream consumers still require dense arrays.

Validation

• make test: passed the full Cairo + native gzip + zlib-ng race suite, plus system-zlib and portable backend race variants.
• go test -race ./pkg/types/...: passed the types and encoding packages.
• Decoder differential fuzzing: passed 613,583 executions with two workers over the 15-second fuzz run.
• CI-pinned golangci-lint 2.14.0: zero issues.
• git diff --check: passed.

Production validation

Tested commit 6c51800 on one production frontend serving normal client traffic, with five unchanged sibling frontends as simultaneous controls. The sequence was 10 minutes stock, 5 minutes candidate warmup, 15 minutes candidate measurement, 5 minutes restored-stock warmup, and 10 minutes restored-stock measurement. The compiler (Go 1.27.0, linux/amd64, GOAMD64=v1), native build tags (cairo,nativegzip,zlibng), configuration and runtime environment matched the incumbent. CPU profiles ran outside the measurement windows.

The candidate measurement completed 373,665 client renders and 615,817 upstream renders. There were 4 render 5xx responses (0.0011%), all in one backend TCP dial-timeout burst; unchanged siblings also recorded 5xx errors during that period. There were no unexpected process restarts or health-guard trips. The host was restored to stock and normal routing; the temporary override and canary controls were removed after evidence collection.

The table uses each complete window's scraped counter deltas. CPU and allocation cost include all service work, divided by completed client renders, so request mix still matters. Latency comes from merged nginx histogram buckets, not averaged host quantiles.

Metric	Initial stock	Candidate	Restored stock
CPU / client render, ms	10.88	10.62	10.94
Allocated MiB / client render	3.002	2.870	3.004
nginx p99, ms	188	228	225
nginx p999, ms	821	1170	1041

Accounting for rendered point volume and simultaneous sibling movement, candidate CPU per point changed -4.4% versus the initial stock window and -9.3% versus restored stock; allocated bytes per point changed -7.3% and -12.0%, respectively. These are descriptive results from one short canary, not a confidence interval or a matched-query replay.

Using only the same-datacenter sibling as the control gives CPU-per-point changes of -0.9% and -8.8%, and allocation-per-point changes of -5.9% and -16.3%, against the two reference windows. Allocation savings are more consistent than the small CPU differences.

Live profiles support the targeted effects: mergeMetrics CPU share was 9.0% / 4.2% / 9.1% across stock / candidate / restored stock, and the decoder's bools allocation share was 6.6% / 1.6% / 6.4%. Final dense point materialization remained about 18% of candidate allocations. The profile shares came from different traffic intervals and should not be read as an exact whole-service speedup.

The profiles support lower merge cost and less allocation. A reliable whole-service CPU or p99/p999 improvement remains unproven: backend timeout bursts and changing query mix affected the fleet during the candidate window. RSS also fluctuated substantially and process restarts confound footprint comparisons; no sustained RSS reduction is claimed.

Rollout and limits

The production canary has been rolled back. A longer matched evaluation is needed before claiming a tail-latency win or extrapolating the resource savings to the fleet. Large queries still fetch and materialize their input series; this PR reduces processing overhead without imposing a query cost limit. Reverting the change restores the previous decoder and merge loops.

Fixes #585

[timtofan]

Closing for now. The live canary showed modest allocation savings, but no proven p99/p999 improvement. This micro-optimisation does not justify pursuing further at present. The measurements remain in the PR for reference; production was restored to the stock build after the test.