[timtofan]

Why

Closes #581.

Render requests with several targets fetch each target's backend data in sequence. Network waits add across targets even though their input reads are independent. Shared missing inputs can also trigger repeated fetches within the same request. This inflates tail latency for multi-target queries.

Architecture

Parse the targets once and submit their distinct static metric reads through the existing worker queues. A request-scoped future is keyed by the complete metric request, including its time bounds. The handler consumes those futures and evaluates expressions in the original target order, retaining shared input storage and the existing error precedence. Cancellation drains unused futures before access logging reads their counters.

The overlap path applies when resolveGlobs: 0 sends globs directly to stores. Expanded-glob mode retains demand scheduling because one expression can produce many individual reads. Transport selection, replica merging, cache keys, and batching policy stay at their existing boundaries.

Before / after

Before	After
Target B starts its static input reads after target A finishes evaluation.	Independent input reads overlap through the existing bounded worker queues.
A missing shared input can be fetched again by a later target.	One request-scoped future retains its response, including missing-data results.
Expressions evaluate and return results in target order.	Expressions evaluate and return results in the same order.
Earlier target errors take precedence over later parse errors.	The same first-target error takes precedence.

Usage

Existing resolveGlobs: 0 deployments activate overlap for requests containing multiple target parameters; single-target requests and expanded-glob deployments retain demand scheduling. A typical consumer is a dashboard panel that requests several related error-rate series in one render call.

Validation

• Full Go race suite passed across 49 packages. Native Go 1.27 application tests passed with cairo,nativegzip,zlibng.
• Differential tests cover generated expressions, shared inputs, empty inputs, fallback behavior, time shifts, result order, and earlier backend errors preceding later parse errors.
• Tests drive the real worker queues to verify fetch overlap and cancellation liveness, including one-worker cancellation and draining an unused request after an earlier backend failure.
• Three replay passes over 69 captured query shapes produced 276 byte-identical baseline/candidate responses per pass, alternating gzip.
• golangci-lint 2.15.0 reports no issues in the changed application package.

A guarded production trial compared two warmed ten-minute windows, totaling 500,001 renders, with three unchanged control processes. Multi-target requests were about 5.2% of traffic. Their raw mean latency decreased from 86.17 to 39.52 ms and p99 from 527.34 to 362.76 ms; query mix can differ between windows.

The whole-workload control-adjusted application histogram ratios were p95 0.628, p99 0.929, and p999 1.029. Approximate two-minute block bootstrap 95% intervals were 0.559–0.780, 0.582–1.437, and 0.783–1.341 respectively. This supports a p95 gain and does not establish an overall p99 or p999 gain. A late candidate window had 33 backend failures versus zero in the reference window, including single-target requests; whether overlap contributed through resource pressure is unresolved. RSS median was approximately 15% higher, with substantial variation. A later ten-minute stock-only observation had 15 render server errors; 14 mentioned backend dialing and 13 mentioned I/O timeouts. This reproduces the error class without prefetch, but the observation is not a controlled comparison and does not rule out additional pressure from overlap. A subsequent stock-only event capture recorded 1,676 backend HTTP-client errors in five seconds; these count individual backend attempts rather than render responses. The following recovery profile averaged five CPU cores and did not show a concentrated transport lock wait. The transient failure mechanism remains unresolved.

The managed process and configuration were restored and verified. The patch is ready for code review for the measured multi-target improvement. Overall p99/p999 benefit is unproven, and the connection-error burst still needs investigation before a broader production rollout.

CI exceeded its existing ten-second package deadline twice in TestStreamingRenderMatchesBuffered; the changed application package passed both times. A separate test-only commit raises race package deadlines to sixty seconds while retaining the new driver's short cancellation and liveness deadlines. Fresh CI passed on Go oldstable, stable, and tip. The stable job also passed vulnerability scanning, lint, and integration tests.

Review questions

What issue is this change attempting to solve? Independent backend waits add across top-level targets; see Why and #581.

How does this change solve the problem? Why is this the best approach? It overlaps only backend input reads through the existing worker queues, retaining sequential evaluation and the current transport and merge boundaries.

How can we be sure this works as expected? Differential, replay, race, and real-driver liveness checks are listed above. The production result and its limits are explicit; broader deployment is not proposed by this PR.