From ce28571fd3358068074d5bca59240c1636e24719 Mon Sep 17 00:00:00 2001 From: susiejojo Date: Mon, 16 Mar 2026 16:35:45 -0400 Subject: [PATCH 1/3] docs: add experiment design justification for ~50-experiment benchmark matrix Explains why these specific models, GPUs, workloads, and config knobs were chosen for the BLIS validation experiments. Covers pruning rationale, figure-driven mapping, phase ordering, Vidur overlap, and ServeGen workload profiles. Written for executive/reviewer audience. Co-Authored-By: Claude Opus 4.6 --- paper/experiment-design-justification.md | 131 +++++++++++++++++++++++ 1 file changed, 131 insertions(+) create mode 100644 paper/experiment-design-justification.md diff --git a/paper/experiment-design-justification.md b/paper/experiment-design-justification.md new file mode 100644 index 0000000..5309f60 --- /dev/null +++ b/paper/experiment-design-justification.md @@ -0,0 +1,131 @@ +# Why These ~50 Experiments? + +BLIS predicts LLM inference latency without GPUs. These experiments validate that prediction across the configurations production operators actually deploy. + +--- + +## The Question We're Answering + +**Can BLIS replace GPU benchmarking for capacity planning?** To answer yes, we need ground-truth measurements spanning diverse models, hardware, workloads, and serving configurations. The matrix is designed to be *minimal* (every experiment maps to a figure) and *sufficient* (covers the axes that matter). + +--- + +## What We Vary (4 Axes) + +### Models (8) — architecture diversity, not model zoo padding + +We pick along two axes that change the simulator's code paths: **dense vs. MoE** and **parameter scale**. + +| Category | Models | What it tests | +|---|---|---| +| Small dense (TP=1) | Llama-2-7B, Llama-3.1-8B, Qwen3-14B | Roofline FLOPs/bandwidth estimation — no parallelism confounds | +| Medium dense (TP=2) | CodeLlama-34B | Tensor-parallel communication overhead | +| Large dense (TP=4) | Llama-2-70B | TP scaling limits (GQA with 8 KV heads) | +| MoE (sparse activation) | Mixtral-8x7B, Mixtral-8x22B | Sparse compute vs. full weight memory footprint | +| MoE + FP8 | Llama-4-Scout-17B-16E | Quantized weights + 16-expert topology | + +**Vidur overlap (deliberate).** Three of our models — Llama-2-7B, Llama-2-70B, CodeLlama-34B — are shared with [Vidur](https://github.com/microsoft/vidur) (Microsoft, NSDI '25), which supports {Llama-2-7B, Llama-2-70B, CodeLlama-34B, Llama-3-8B, Llama-3-70B, InternLM-20B, Qwen-72B} on {A40, A100, H100}. This overlap enables direct apples-to-apples accuracy comparison on the same model×hardware pairs. Our matrix then extends beyond Vidur's coverage with MoE models (Mixtral-8x7B, Mixtral-8x22B, Llama-4-Scout), FP8 quantization, expert parallelism, L40S hardware, and config knob sweeps — none of which Vidur evaluates. + +**Excluded:** Vision-language models (different compute graph), 405B+ (same code path as 70B, 4× the GPU cost). + +### Hardware (3) — bandwidth tiers that exist in datacenters + +| GPU | Bandwidth | Role | +|---|---|---| +| H100-80GB | 3.35 TB/s, NVLink | Primary — most experiments here | +| A100-80GB | 2.0 TB/s, NVLink | Cross-generation validation (measured 1.6-1.7× slower than H100, matching the bandwidth ratio) | +| L40S-48GB | 864 GB/s, PCIe | Budget tier — tests viability limits | + +Three GPUs = three distinct (bandwidth × interconnect) classes. Adding H200 or A10G covers no new class. + +### Workloads (4) — from ServeGen traffic profiles + +These four workloads are drawn from [ServeGen](https://arxiv.org/abs/2401.09056) traffic profiles, each representing a structurally distinct serving regime: + +| Workload | Input | Output | Traffic pattern | What it stresses | +|---|---|---|---|---| +| General | 200-800 | ~250 | Highly variable burstiness, temporal shifts | Auto-scaling, dynamic resource allocation — tests the simulator under non-stationary arrival rates | +| Codegen | 500-1500 | ~250 | Development-cycle patterns, template-based outputs | Structured generation with long prefills — stresses chunked-prefill batch formation | +| Roleplay | 100-400 | ~250 | Smoother, human-paced interaction | Conversational serving with steady load — baseline for prefill/decode balance | +| Reasoning | 934 | ~1448 | Bimodal output (reasoning + answer tokens) | Long-output decode with 6× longer batch occupancy — stresses KV cache capacity and preemption | + +Together they cover the (input_length, output_length, burstiness) space: short-input steady (Roleplay), long-input bursty (General, Codegen), and long-output bimodal (Reasoning). This ensures the simulator is validated across the traffic shapes that actually differ in how they load prefill vs. decode and how they pressure KV cache memory. + +### Config Knobs (5) — what operators actually tune + +`max_num_batched_tokens` (1K/2K/8K), `cpu_offloading` (on/off), `gpu_memory_utilization` (0.9/0.95), tensor parallelism (default/2×), expert parallelism (MoE only: 1/2/4). + +--- + +## Why ~50, Not Thousands? + +Full factorial = 8 × 3 × 4 × many knob combos = thousands. Three pruning rules get us to ~50: + +1. **Viability pruning.** Large models don't fit on small GPUs. Validated with capacity formulas → 18 viable model×GPU combos (down from 24). + +2. **Config sweeps on 2 models only.** Knob sensitivity depends on architecture class (dense vs. MoE), not model identity. We sweep on the cheapest model per class: Llama-3.1-8B (1 GPU) and Mixtral-8x7B (2 GPUs). Sweeping Llama-2-70B instead would cost 4× the GPU-hours for the same information. Saves ~70 experiments. + +3. **Cross-hardware = General workload only.** Hardware scaling is workload-independent (confirmed: A100/H100 ratio is 1.6-1.7× regardless of workload). Running all 4 workloads on A100 and L40S would be redundant. Saves ~21 experiments. + +--- + +## Every Experiment Maps to a Figure + +| Figure | Claim | Experiments | Key comparison | +|---|---|---|---| +| **Fig 1: Model comparison** | Accurate across 7B-141B, dense + MoE | 7 | Architecture is sole variable | +| **Fig 2: Hardware comparison** | Bandwidth-based roofline generalizes across GPUs | ~16 | Same model, different GPU | +| **Fig 3: Workload comparison** | Predictions hold across token distributions | ~28 | Same model+GPU, different workload | +| **Fig 4: Config sensitivity** | Captures impact of operator-tunable knobs | 13 | Same model, one knob varied | + +Zero experiments exist outside these four figures. + +--- + +## Phase Ordering: Publishable at Any Stopping Point + +GPU time is expensive. The matrix is ordered so you can stop early and still have results. + +| Stop after | Experiments | What you have | +|---|---|---| +| Phase 0-3 | 21 | 5 models × 3 workloads on H100 (Figures 1, 3) | +| Phase 0-5 | 32 | + config sweeps (Figure 4 complete) | +| Phase 0-8 | 44 | + A100/L40S cross-hardware (Figure 2 complete) | +| Phase 0-9 | ~50 | + Reasoning workload (full matrix) | + +--- + +## Measurement Rigor + +- **Sub-saturation by design.** Every experiment runs below 50% of calibrated throughput limit. At saturation, latency is dominated by queueing (tests the scheduler, not the latency model). Sub-saturation isolates what we're actually validating. +- **Single-variable comparisons.** Each experiment varies one dimension from a per-model baseline (default TP, mbt=2048, no offload, gpu_mem=0.9). +- **Stock vLLM v0.15.1.** No custom fork. Reproducible by anyone. +- **Pre-calibration.** Every model×GPU×TP combo has a measured per-token decode latency. Experiments that exceed the safe rate are flagged and rate-capped. + +--- + +## vs. Prior Work + +| System | Models | GPUs | Workloads | Config sweeps | MoE | FP8 | +|---|---|---|---|---|---|---| +| Vidur (NSDI '25) | 3 dense | 1 (A100) | 2 | No | No | No | +| Splitwise (ISCA '24) | 2 dense | 1 (A100) | 1 | No | No | No | +| DistServe (OSDI '24) | 3 dense | 2 (A10G, A100) | 2 | Partial | No | No | +| **This work** | **5 dense + 3 MoE** | **3 (H100, A100, L40S)** | **4** | **5 knobs, 11 sweeps** | **Yes** | **Yes** | + +Three of our dense models (Llama-2-7B, Llama-2-70B, CodeLlama-34B) and two GPUs (A100, H100) overlap with Vidur, enabling head-to-head accuracy comparison on shared configurations. Our matrix then extends into territory no published simulator evaluation covers: MoE architectures, FP8 quantization, expert parallelism, and systematic config knob sensitivity. + +--- + +## Known Limitations + +- **vLLM only** — doesn't validate against TensorRT-LLM or SGLang (different schedulers, future work) +- **Single-node** — max 8 GPUs, no pipeline parallelism across nodes +- **Synthetic workloads** — controlled token distributions, not production traces (by design: isolates latency model accuracy from arrival process noise) +- **FP8 only for MoE** — reflects current practice where FP8 is used to fit models that wouldn't otherwise be viable + +--- + +## Bottom Line + +~50 experiments. 8 architectures (7B-141B, dense + MoE + FP8). 3 GPU tiers. 4 workloads. 5 serving knobs. Every experiment maps to a figure. Publishable results at any early-stopping phase. Broader than any published simulator evaluation we're aware of. From e72759b1f40caa9a1d4565aac0f312fbc7aa5eb8 Mon Sep 17 00:00:00 2001 From: susiejojo Date: Mon, 16 Mar 2026 16:39:22 -0400 Subject: [PATCH 2/3] fix: correct ServeGen link to github.com/alibaba/ServeGen Co-Authored-By: Claude Opus 4.6 --- paper/experiment-design-justification.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/paper/experiment-design-justification.md b/paper/experiment-design-justification.md index 5309f60..a897464 100644 --- a/paper/experiment-design-justification.md +++ b/paper/experiment-design-justification.md @@ -40,7 +40,7 @@ Three GPUs = three distinct (bandwidth × interconnect) classes. Adding H200 or ### Workloads (4) — from ServeGen traffic profiles -These four workloads are drawn from [ServeGen](https://arxiv.org/abs/2401.09056) traffic profiles, each representing a structurally distinct serving regime: +These four workloads are drawn from [ServeGen](https://github.com/alibaba/ServeGen) traffic profiles, each representing a structurally distinct serving regime: | Workload | Input | Output | Traffic pattern | What it stresses | |---|---|---|---|---| From f1f44c98f114f68aa31a2f2aaa73405fd92275eb Mon Sep 17 00:00:00 2001 From: susiejojo Date: Mon, 16 Mar 2026 16:45:26 -0400 Subject: [PATCH 3/3] docs: link Discussion #598 experiment matrix from justification doc Co-Authored-By: Claude Opus 4.6 --- paper/experiment-design-justification.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/paper/experiment-design-justification.md b/paper/experiment-design-justification.md index a897464..3dc6be7 100644 --- a/paper/experiment-design-justification.md +++ b/paper/experiment-design-justification.md @@ -1,6 +1,6 @@ # Why These ~50 Experiments? -BLIS predicts LLM inference latency without GPUs. These experiments validate that prediction across the configurations production operators actually deploy. +BLIS predicts LLM inference latency without GPUs. These experiments validate that prediction across the configurations production operators actually deploy. Full experiment matrix: [Discussion #598](https://github.com/inference-sim/inference-sim/discussions/598). ---