Reduce Datadog allocation for small batches

[Areson]

Why

The Datadog sink reserves space for 10,000 series even when a batch contains only a few dozen metrics. These short-lived buffers create avoidable allocation and garbage collection work.

What

Size initial series, cursor, offset, and raw JSON buffers from available input. Keep the 10,000-series conversion ceiling and exact payload limits unchanged.

How

Use remaining source slice lengths to bound series and cursor capacity. Estimate raw capacity from at most eight evenly spaced series and sixteen tags per sampled series. The estimate uses string lengths and an upper median, with a 512-byte floor per series and overflow-safe caps. It adds no full conversion pass or retained buffers. Serialization, gzip settings, chunking, checkpoints, and retry behavior stay unchanged.

Risk

This affects every Datadog send. A bounded estimate can miss large fields or JSON escaping: allocation still exceeds baseline for some inputs. The measured minority-large case regresses 42.85%; the escape-heavy case regresses 18.03%. This is not a universal memory or latency improvement.

Testing

Compared baseline a5c9f4b, initial sizing dbcc2b7, and final 6a99d5c2 using identical inputs and a test-only consumer dependency graph. Measurements used Go 1.26.4, Datadog SDK 2.37.1, CGO disabled, GOMAXPROCS=2, GOGC=100, and GOMEMLIMIT=off. This PR does not change dependencies.

Six balanced-order runs used ten fixed sends per case through SendWithCheckpoint and the SDK's CallAPI. Native Linux arm64 median allocated bytes per send:

Synthetic batch	Baseline B/op	Final B/op	Change
61 ordinary series	7,323,186	960,101	-86.89%
383 ordinary series	7,825,985	1,683,095	-78.49%
1,000 series, compressible 4 KiB tags	23,404,586	21,890,262	-6.47%
1,000 series, poorly compressible 4 KiB tags	113,695,460	103,587,818	-8.89%
1,000 series, 16 KiB tags on every fourth series	22,704,772	32,434,285	+42.85%
1,000 series, escape-heavy tags	67,796,434	80,023,428	+18.03%

The local comparison harness ran -test.run '^$' -test.bench '^BenchmarkRefinement$' -test.benchtime=10x -test.count=1 for each variant and order. It covered 26 distributions, including mixed sizes, early/late outliers, Unicode, many tags, and large names. Linux amd64 runs used emulation; their CPU and memory results do not qualify native amd64 performance.

Committed benchmarks can be run with go test ./sink -run '^$' -bench '^BenchmarkDatadogCapacity' -benchmem -benchtime=10x -count=6. They use the repository's dependencies by default, so their absolute results differ from the matched consumer comparison above.

• 72 local three-way comparisons matched raw/compressed bytes, hashes, chunk counts, indices, and size-limit errors. Cases included nonzero starts and exact soft/hard boundaries.
• go test -race ./sink -count=1 passed on native macOS arm64 with both upstream and matched consumer dependencies. The upstream race run was refreshed before PR creation.
• A local two-monitor replay at 100m CPU and 178 MiB acknowledged all 228,380 offered series in all three variants. Each made 390 requests with identical wire bytes and two timeout events, then recovered. Baseline/final peaks were 116.77/110.30 MiB; post-idle live heap was about 7.1 MiB for both. Earlier ten-minute baseline/initial runs failed the strict no-backlog gate and had higher peaks after the initial patch; recovery does not establish timeout-free operation.
• An anonymized two-instance staging comparison measured runtime allocation at 15.254 versus 2.553 MB/s, about 83.3% lower. Workloads were nearly matched but differed in metric-value count; this observation is not a controlled universal guarantee.

Generated with Codex