Repository navigation
[Benchmark] 2× RTX 5070 Ti (PCIe 5.0 x8 each) + Ryzen 9 9950X + 128 GB DDR5-6000 | Qwen3.8-Flash-Next IQ3_S and UD-IQ4_XS | 262K context | Strata 0.2.7 source build #4418
Description
Two runs of strata bench run (standard-v1) on the same machine with the 0.2.7 tag built from source (engine 0.1.52, CUDA 13.2, sm_120): IQ3_S and Unsloth UD-IQ4_XS, both split across the two cards (--gpus 0,1), context 262144, settings exactly as written by setup.py. Both runs pass 12 of 12 sanity checks.
| Model | Short decode tok/s | 4K prompt tok/s | 16K prompt tok/s | 16K decode tok/s | Sanity |
|---|---|---|---|---|---|
| IQ3_S | 129.4 | 2322.0 | 3823.4 | 116.2 | 12/12 |
| UD-IQ4_XS | 81.9 | 1896.0 | 3734.2 | 87.6 | 12/12 |
For context, an informal before/after on this machine (my own 600-token request, not the standard suite): the same setup on 0.1.42 gave about 87-101 tok/s decode for IQ3_S and about 76 tok/s for UD-IQ4_XS; on 0.2.7 the same request gives about 115-122 and 74-87 tok/s. The two cards are a Gen5 x8/x8 split; card 0 also drives the desktop.
Full report: IQ3_S
Strata standard benchmark: NVIDIA GeForce RTX 5070 Ti, NVIDIA GeForce RTX 5070 Ti
Protocol strata-qwen38-thinking-v1-draft.1 (draft), run 20261027T230845Z-c325695e, started 2026-10-27T23:08:45Z. Status: complete.
Written by strata bench run. The machine-readable result is results.json; every request, its raw usage and timings and the client event offsets are in outputs.jsonl.
Hardware and software
- GPU: NVIDIA GeForce RTX 5070 Ti (15.9 GiB dedicated); NVIDIA GeForce RTX 5070 Ti (15.9 GiB dedicated)
- CPU: AMD Ryzen 9 9950X 16-Core Processor; RAM: 125.4 GiB
- Storage of the model files: nvme CT2000T500SSD8 (nvme)
- OS: Ubuntu 24.04.5 LTS 7.0.0-38-generic; drivers: gpu-driver 595.99.02; runtime: cuda 13.2
- Strata commit:
5e8a4cca7d690ac60ca37eb09e7d0e010099ad6d; engine 0.1.52; source build - Background work: not measured
Model and configuration
- Model: qwen3.8-flash-next-iq3_s; quantization IQ3_S; revision not measured; source revision not measured
- Files:
Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf4c1eb2ceb4915e11...;Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf316b46f3a2dbd68c...;expert-profile.bin8f59b4aa8873209d... - Track: custom (no notes given by the operator)
- Context 262144 tokens; server capacity 1 slot(s)
- Sampling: temperature 1.0, top_p 0.95, top_k 20, min_p 0.0, presence_penalty 0.0, repetition_penalty 1.0, reasoning_effort xhigh; thinking on, preserved
- Engine options:
config.json; chat template:provenance/chat_template.jinja
Method
The frozen requests of bench/prompts/standard-v1 (corpus e75eff8b9feea04f...) were sent over loopback HTTP, streaming, one at a time, in this order: 1 warm-up (excluded), 3 repetitions of the short, 4K and 16K prompts, and the twelve sanity checks. Prompt and decode speed are the engine's own timings; time to first token is the client's monotonic clock from dispatch to the first nonempty reasoning, answer or tool-call delta. No request was retried. Time after the model loaded: 1 min 18 s.
Results
| Case | Prompt tok/s | Decode tok/s | TTFT s | First answer s | Total s | Capped |
|---|---|---|---|---|---|---|
| short | n/a | 129.4 (122.2-131.5) n=3 | 0.55 (0.46-0.55) n=3 | 1.36 (1.25-1.39) n=3 | 3.1 (2.8-3.2) n=3 | 0/3 |
| prompt-4k | 2322.0 (2269.5-2323.5) n=3 | 124.3 (124.0-139.3) n=3 | 1.79 (1.78-1.82) n=3 | 4.55 (3.12-4.73) n=3 | 5.9 (5.3-5.9) n=3 | 2/3 |
| prompt-16k | 3823.4 (3797.4-3840.0) n=3 | 116.2 (115.3-127.1) n=3 | 4.30 (4.29-4.33) n=3 | 7.19 (7.19-7.19) n=1 | 8.7 (8.3-8.7) n=3 | 3/3 |
Cells are median (min-max) n=samples. Short is a prompt of about 80 tokens: its prompt rate is dominated by the fixed cost of a request, so it is not shown; read prompt speed from prompt-4k and prompt-16k. Capped = requests that reached the 512-token cap, reasoning and answer together. A capped request's total time is the time to the cap, not to an answer, and it has no first-answer time when it was still reasoning.
| Phase | Planned | Completed | Truncated | Failed | Skipped |
|---|---|---|---|---|---|
| warmup | 1 | 0 | 1 | 0 | 0 |
| performance | 9 | 4 | 5 | 0 | 0 |
| sanity | 12 | 12 | 0 | 0 | 0 |
| concurrency | 0 | 0 | 0 | 0 | 0 |
Sanity checks: 12 of 12.
| Case | Result | Reason |
|---|---|---|
| I1 | pass | the expected JSON value |
| I2 | pass | the expected JSON value |
| I3 | pass | the expected JSON value |
| I4 | pass | the exact expected text |
| R1 | pass | the exact expected text |
| R2 | pass | the exact expected text |
| R3 | pass | the exact expected text |
| R4 | pass | the expected JSON value |
| T1 | pass | the expected tool call |
| T2 | pass | the expected tool call |
| T3 | pass | the expected tool call |
| T4 | pass | the expected tool call |
Failures, exclusions and limitations
- No request failed.
- Excluded from speed summaries: performance-prompt-16k-r1: first_answer_s is not available for this request
- Excluded from speed summaries: performance-prompt-16k-r2: first_answer_s is not available for this request
- Prompt-prefix reuse is the shared template prefix only; the cache was not emptied between requests, and the expert cache stays warm and evolves. These are not cold-model measurements.
- Output caps include reasoning; a request that reaches its cap is
truncatedand fails its sanity check. - The runner cannot see other programs on the machine, the repeat-stop and loop-recovery switches of the server, or the quantization repository's revision; see
missingin results.json for every value it could not read. - The server had answered no request before this run (a freshly loaded engine).
Full report: UD-IQ4_XS
Strata standard benchmark: NVIDIA GeForce RTX 5070 Ti, NVIDIA GeForce RTX 5070 Ti
Protocol strata-qwen38-thinking-v1-draft.1 (draft), run 20261027T231212Z-f6689c3f, started 2026-10-27T23:12:12Z. Status: complete.
Written by strata bench run. The machine-readable result is results.json; every request, its raw usage and timings and the client event offsets are in outputs.jsonl.
Hardware and software
- GPU: NVIDIA GeForce RTX 5070 Ti (15.9 GiB dedicated); NVIDIA GeForce RTX 5070 Ti (15.9 GiB dedicated)
- CPU: AMD Ryzen 9 9950X 16-Core Processor; RAM: 125.4 GiB
- Storage of the model files: nvme CT2000T500SSD8 (nvme)
- OS: Ubuntu 24.04.5 LTS 7.0.0-38-generic; drivers: gpu-driver 595.99.02; runtime: cuda 13.2
- Strata commit:
5e8a4cca7d690ac60ca37eb09e7d0e010099ad6d; engine 0.1.52; source build - Background work: not measured
Model and configuration
- Model: qwen3.8-flash-next-unsloth-ud-iq4_xs; quantization UD-IQ4_XS; revision not measured; source revision not measured
- Files:
Qwen3.8-Flash-Next-UD-IQ4_XS-00001-of-00003.gguf5ce89370720f8bf9...;Qwen3.8-Flash-Next-UD-IQ4_XS-00002-of-00003.gguf577a38a2392b40ca...;Qwen3.8-Flash-Next-UD-IQ4_XS-00003-of-00003.ggufd4634e6d84f0ebb0...;expert-profile.bin8f59b4aa8873209d... - Track: custom (no notes given by the operator)
- Context 262144 tokens; server capacity 1 slot(s)
- Sampling: temperature 1.0, top_p 0.95, top_k 20, min_p 0.0, presence_penalty 0.0, repetition_penalty 1.0, reasoning_effort xhigh; thinking on, preserved
- Engine options:
config.json; chat template:provenance/chat_template.jinja
Method
The frozen requests of bench/prompts/standard-v1 (corpus e75eff8b9feea04f...) were sent over loopback HTTP, streaming, one at a time, in this order: 1 warm-up (excluded), 3 repetitions of the short, 4K and 16K prompts, and the twelve sanity checks. Prompt and decode speed are the engine's own timings; time to first token is the client's monotonic clock from dispatch to the first nonempty reasoning, answer or tool-call delta. No request was retried. Time after the model loaded: 1 min 43 s.
Results
| Case | Prompt tok/s | Decode tok/s | TTFT s | First answer s | Total s | Capped |
|---|---|---|---|---|---|---|
| short | n/a | 81.9 (75.8-89.2) n=3 | 0.67 (0.59-0.69) n=3 | 1.99 (1.20-2.08) n=3 | 4.8 (3.8-5.0) n=3 | 0/3 |
| prompt-4k | 1896.0 (1891.8-1916.0) n=3 | 83.7 (78.9-86.9) n=3 | 2.18 (2.15-2.19) n=3 | 4.59 (4.40-8.24) n=3 | 8.3 (8.0-8.7) n=3 | 3/3 |
| prompt-16k | 3734.2 (3730.7-3757.8) n=3 | 87.6 (81.5-93.8) n=3 | 4.41 (4.39-4.42) n=3 | 6.81 (6.29-7.33) n=2 | 10.2 (9.9-10.7) n=3 | 3/3 |
Cells are median (min-max) n=samples. Short is a prompt of about 80 tokens: its prompt rate is dominated by the fixed cost of a request, so it is not shown; read prompt speed from prompt-4k and prompt-16k. Capped = requests that reached the 512-token cap, reasoning and answer together. A capped request's total time is the time to the cap, not to an answer, and it has no first-answer time when it was still reasoning.
| Phase | Planned | Completed | Truncated | Failed | Skipped |
|---|---|---|---|---|---|
| warmup | 1 | 0 | 1 | 0 | 0 |
| performance | 9 | 3 | 6 | 0 | 0 |
| sanity | 12 | 12 | 0 | 0 | 0 |
| concurrency | 0 | 0 | 0 | 0 | 0 |
Sanity checks: 12 of 12.
| Case | Result | Reason |
|---|---|---|
| I1 | pass | the expected JSON value |
| I2 | pass | the expected JSON value |
| I3 | pass | the expected JSON value |
| I4 | pass | the exact expected text |
| R1 | pass | the exact expected text |
| R2 | pass | the exact expected text |
| R3 | pass | the exact expected text |
| R4 | pass | the expected JSON value |
| T1 | pass | the expected tool call |
| T2 | pass | the expected tool call |
| T3 | pass | the expected tool call |
| T4 | pass | the expected tool call |
Failures, exclusions and limitations
- No request failed.
- Excluded from speed summaries: performance-prompt-16k-r2: first_answer_s is not available for this request
- Prompt-prefix reuse is the shared template prefix only; the cache was not emptied between requests, and the expert cache stays warm and evolves. These are not cold-model measurements.
- Output caps include reasoning; a request that reaches its cap is
truncatedand fails its sanity check. - The runner cannot see other programs on the machine, the repeat-stop and loop-recovery switches of the server, or the quantization repository's revision; see
missingin results.json for every value it could not read. - The server had answered no request before this run (a freshly loaded engine).
Activity
bob-u commentedon Oct 28, 2026
I wonder what makes AMD (HIP) cards so weak in prompt processing compared to Nvidia.
Niko1221 commentedon Oct 29, 2026
Thank you for measuring this and writing it up. We are building a Strata website with a page for community benchmarks, and results like these will be added there when it launches, so we are closing this here. If your numbers change with a new version, you are welcome to post them again.