[Bug]: Preset reported `tps/user` ~10x too low with sglang bench and speculative decoding

[r4victor]

Steps to reproduce

1. Create a preset where the agent picks speculative decoding and benchmarks with sglang bench_serving:

type: preset
name: qwen3-27b-agentic-h100
base: Qwen/Qwen3.8-27B
min_context_length: 262144
max_ttft: 5000
trials: 4
concurrency: 4
input_tokens: 131072
output_tokens: 512
shared_prefix_tokens: 130048
fleets: [h100-fleet]

2. Run dstack preset list.

Which tool and whether speculative decoding wins are the agent's choices, so this may take more than one attempt.

Actual behaviour

The preset is reported as tps/user=7.29. The real decode rate is about 78 tok/s, so the headline metric is ~10x too low, in the pessimistic direction, in units that invite comparison with published figures for the same model.

tps/user is 1000 / tpot_ms.mean, and tpot_ms is whatever the agent puts in the report:

dstack/src/dstack/_internal/core/models/presets.py

Lines 104 to 106 in ccf5d81

	@property
	def effective_per_user_tok_per_s(self) -> float:
	return 1000 / self.metrics.tpot_ms.mean

dstack/src/dstack/_internal/cli/services/presets/output.py

Lines 213 to 216 in ccf5d81

	# Lead with per-user tps: comparable across rows regardless of concurrency.
	tpot_ms = best.get("tpot_ms")
	if isinstance(tpot_ms, (int, float)) and tpot_ms > 0:
	parts.append(f"tps/user={_format_number(1000 / tpot_ms)}")

In my session the agent put the benchmark tool's Inter-Token Latency row there, byte for byte:

tool printed                     report tpot_ms
Mean ITL (ms):    137.18    ->   mean: 137.18
Median ITL (ms):   55.34    ->   p50:   55.34
P99 ITL (ms):    1160.43    ->   p99: 1160.43

It had little choice: tpot_ms is a required field, sglang.benchmark.serving is a sanctioned tool in the system prompt, and that tool prints no TPOT row at all (it computes one and writes it only to --output-file).

ITL is timed per streamed chunk, not per token. Without speculative decoding a chunk is one token and ITL equals TPOT, which is why this has gone unnoticed. With speculative decoding a chunk carries all accepted draft tokens, so ITL is inflated by the accept length. Here that was num_speculative_tokens 15 with roughly 10 tokens per chunk.

Nothing detects it, although the report contradicts itself. The same object holds duration_seconds: 96.28, total_output_tokens: 16384, concurrency: 4 and tpot_ms.mean: 137.18. A mean TPOT of 137.18ms implies 2243 seconds of decode work, against the 385 slot-seconds that four slots over 96.28s can supply. Aggregate throughput cannot exceed concurrency times the per-user decode rate, and this report exceeds it by 5.8x.

Note that effective_output_tok_per_s immediately above deliberately recomputes from totals rather than trusting the agent's self-reported rate. effective_per_user_tok_per_s is the one metric that does not.

The blast radius is not narrow: speculative decoding is the winning configuration on every recent preset I have run, on both AMD and NVIDIA hardware, so the affected case is the common one rather than the exotic one.

Expected behaviour

tps/user reflects the measured per-token decode rate, and a report whose latency figures contradict its own totals is rejected rather than stored.

Solution

Derive TPOT from the report's own totals instead of accepting it from the agent, the way effective_output_tok_per_s already does:

tpot_ms = (e2e_ms.mean - ttft_ms.mean) / (avg_output_len - 1)

This needs one new required field, e2e_ms. Both sanctioned tools print it, Mean E2EL in vllm bench serve and Mean E2E Latency in sglang bench_serving, unlike TPOT.

Then keep tpot_ms stored as reported and never read back, matching output_tok_per_s and per_user_tok_per_s, and update the ## Benchmark section of the system prompt so the agent supplies e2e_ms rather than a TPOT its tool may not print.