Add multi-process KLL vs exact streaming demo with Prometheus and Grafana

[zzylol]

Adds persistent, mergeable KLL panes with rolling quantiles, and a live demo that runs KLL next to exact on the same traffic, with every processor in its own pinned process, reporting throughput, CPU and accuracy to Prometheus/Grafana.

Before → after

• Before: the finite demo wrote estimates to console and files, then exited.
• After:
  • The finite demo can export its p50/p99 to Prometheus.
  • KllWindowStore / window_slide retain tumbling KLL panes and merge them for aligned range and rolling queries.
  • demos/kll-grafana/run-demo.sh runs the exact and KLL pipelines side by side until Ctrl+C. Branch A/B, merge and estimate are separate processes (one container each, --cpuset-cpus to one CPU), connected by OTLP/HTTP over TCP. This is the asap-stream-bench topology from Benchmark sustained OTLP streaming and real network resource use #19, so the demo and the benchmark measure the same code.
  • Grafana shows validated throughput, CPU-seconds per million signals, per-process CPU/egress/RSS, tumbling and rolling p50/p99 against the exact reference, and relative error.

Run

demos/kll-grafana/run-demo.sh                      # Ctrl+C stops
demos/kll-grafana/run-demo.sh --demo-rate 100000   # signals/s per source

Open http://localhost:3000/d/asap-kll-stream. Linux + Docker Compose; each pipeline needs 7 CPUs (2 generator, 4 processors, 1 backend).

56-core x86 server, 50,000 signals/s per source. KLL keeps up with 100,000 signals/s at 0.96 processor CPU-seconds per million signals. Exact uses 23 and is back-pressured to about 68,000 signals/s.

Throughput: KLL vs exact (vs raw forwarding)

Same harness as #19: 65,536-observation windows, 3 runs per point, one dedicated CPU per processor, a 56-core x86 server. Median backend-validated signals/s, both sources:

Offered	Raw	Exact	KLL
20,000	21,825	21,797	21,800
50,000	48,025	47,962	47,992
100,000	100,281	61,038	100,249
200,000	200,542	61,033	200,486
400,000	296,472	61,042	396,609
600,000	300,830	52,322	601,453
800,000	305,168	47,961	797,614

By the CAPACITY_DESIGN criteria, the largest sustainable offered rate per source is 10,000 for exact, 100,000 for raw, and at least 400,000 (the highest tested) for KLL.

Processor CPU and egress summed over branches, merge and estimate, at the same load:

Offered per source	Exact CPU	KLL CPU	Exact egress	KLL egress
10,000	0.63 cores	0.031 cores	36.7 Mbit/s	0.1 Mbit/s
25,000	1.41 cores	0.063 cores	84.4 Mbit/s	0.2 Mbit/s

Accuracy

The backend validates every window: exact must match, KLL must be within 5%. Observed KLL relative error: p50 0.5%, p99 1.6%, tumbling and 6-window rolling.

Validation

python3 demos/kll-grafana/verify-streaming.py   # 60 s; checks metrics, accuracy and Grafana queries
cargo test --features otap-engine,stream-benchmark   # includes rolling-window model tests

Library, window, runtime, and benchmark-model tests pass, along with fmt and Clippy. The demo ran end to end on two servers. Stopping with Ctrl+C or SIGTERM removes all containers and networks.

Limits: count-aligned windows replaying one deterministic corpus (so quantile lines are flat by design); in-memory retention; KLL value error, not rank error. See the demo and window guide.