Repository navigation
gfx11: sync with master (2026-10-16) - #151
Merged
Merged
Conversation
* agents: add note about skipping forks * rm critical line
* add windows vulkan arm64 release * add link
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
…#29806) * ggml-cpu: vectorize BF16 K tails in tinyBLAS * tests: Skip tinyBLAS when use_ref is enabled so CPU tests compare against the vec_dot path. * ggml-cpu: vectorize tinyBLAS F16/F32 tails
* CUDA: refactor swizzling code * fix templates/loop bounds
* llama: support both embd + raw tokens in batch * add to test-llama-archs * also check case llm_arch_supports_mixed_batch = false * constant graph topology * have dedicated input for mixed case * rm set_tensor_backend * is_embd --> type * consolidate m-rope pos handling into one place * nits
… router mode (ggml-org#29895) * log, server: make router child lines carry their own colors The logger writes the color reset after the trailing newline, so the reset opens the next line. On the shared pipe of a router child it lands in front of the next state command, which the router then misses, and the line break that works around it shows up as an empty log line on every progress update. The reset now goes before the trailing newlines, so every line is self contained and the command goes back to its plain framing. The router passes its effective color setting to its children, whose output ends up in its terminal, and leaves that option out when comparing presets on reload. * log: enable ANSI colors on the Windows console A Windows console renders ANSI sequences only in virtual terminal mode, which nothing turns on for the logger, so llama-server prints raw escape codes on the Windows 10 console while llama-cli, whose console code enables it, shows colors. The logger now enables virtual terminal mode on stdout and stderr when it turns colors on, and keeps colors off when a console cannot render them. Pipes and files take the sequences as is. * server: separate the router child commands from its logs The child sent its state commands on the same pipe as its logs, so the router had to pick them out of the log stream by a line prefix, and any unterminated write in front of a command made the router miss it. This resolves the TODO at the spawn that called for splitting stdout and stderr. The child now keeps stdout for the commands and points everything else written to stdout at stderr, before anything is written. The router reads both pipes, handles the commands from stdout and forwards stderr as the log, and warns about any other line on the command pipe. * server: address review from ngxson The single server_child is now created first in the entry point and its constructor keeps stdout for the commands, so the stream is a member of the instance instead of a static, and init() is gone. The instance is passed down to the server, while the CLI entry point creates its own. * Update tools/server/server.cpp --------- Co-authored-by: Xuan-Son Nguyen <thichthat@gmail.com>
…g#29633) * CUDA: use MMVF for thin f16/bf16 mul_mat at small batch size Signed-off-by: ynankani <ynankani@nvidia.com> * adjust kernel selection logic --------- Signed-off-by: ynankani <ynankani@nvidia.com> Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
* metal : few-row MMA mat-mul and batched copies for speculative decoding Speculative decoding verifies a few draft tokens per step. Without the tensor API, Metal ran these mat-muls with the mat-vec kernels, whose time grows with every src1 row, so DFlash2 decoding on an M3 Ultra was slower than serial decoding. - add mat-mul kernels for 2..16 src1 rows on 8x8 simdgroup matrices: each weight is dequantized once for all rows, and the simdgroups of a threadgroup split K. Q4_0, Q8_0 and Q5_K have their own kernels, F32, F16, Q4_1, Q5_0, Q5_1, Q4_K and Q6_K use a generic path over the 16-weight dequantizers, and Q4_0 at 2 rows uses a 2-row variant of the mat-vec kernel - use them only on MTLGPUFamilyApple7+ without the tensor API, from the row count at which they beat the mat-vec kernels on an M3 Ultra (F32: 6, F16, Q4_K, Q5_0, Q5_1: 3, other types: 2) - fusion table: MUL_MAT + ADD adds a same-shape residual in the MMA store, and up to 16 adjacent same-layout f32 copies between the same two tensors run as one dispatch - the fusion checks and ggml_graph_optimize take the device props, so the reorder packs MUL_MAT + ADD only on devices that can fuse it, at every src1 row count - views do not count toward GGML_METAL_FUSION_MAX when the reorder packs a group, so 16 recurrent state snapshot copies with views between them stay one group - the encoder checks the inner nodes of a fused group for concurrency, tracks written views by their extent, and does not count the destination of a CPY as a read - CONCAT splits long rows across threadgroups when there are few rows - tests: few-row MUL_MAT, MUL_MAT_ADD, CPY_BATCH and CONCAT cases in test-backend-ops (with a prepare_graph hook for the copy order), test-metal-graph-optimize, test-metal-cpy-batch-alias * metal : remove the CPY_BATCH fusion and the memory range changes Remove the batched copy fusion with its kernel and tests, and revert the memory range changes, as suggested in review. The memory ranges, the graph reorder and the CPY encoder are again the same as on master. * cont : clean-up * cont : drop has_tensor gate * cont : clean-up operand/residual logic * cont : drop Q4_0 ne11=2 special-case * cont : add kernels/mul_mv_mma.metal * cont : consolidate mma pipeline selection logic * cont : decouple fusion logic from device props --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
…ml-org#29901) * cuda: tile the lightning indexer over keys and tokens for 4 heads With too few heads for a wmma tile, a block scores 64 keys against 8 tokens: the keys are staged once in half precision, the queries one head at a time, and each thread owns one key for two tokens, so no dot product needs a cross thread reduction. Batches smaller than a token tile keep the vector kernel. test-backend-ops measures 4 heads. * cuda: multiply the lightning indexer tile in float Address review from am17an: the half2 products overflow once a single q * k exceeds the f16 range. The queries stay in float in shared memory and each half2 of keys is widened once for both tokens, so every product and sum is computed in float. * cuda: widen each lightning indexer key once for all heads The tile kernel stages the queries and weights of every head at once, so each key element is widened from half once and feeds all heads, with a single barrier. F16 keys are copied into the tile without a float round trip. Keeping the keys in float in shared memory measures slower, the occupancy drops. * cuda: stop the lightning indexer tile from spilling registers on ROCm Each thread of the tile kernel now scores two keys for a single token, so a warp shares its token and the query reads are broadcasts: six shared reads per element pair instead of nine for the same products. The inner loop is unrolled by 8, which keeps gfx908 at 63 VGPRs with no spill where the fully unrolled loop needed over a thousand, and makes the kernel 36x faster on an R9700 and slightly faster on CUDA.
…l-org#29483) * add supports for q1/q5/q3_k/q5_k/q6_k/mxfp4 of mmvq path * Add K_QUANTS_HANDLING macro to q1_0 of mmvq path
* ggml-cpu : add Q8_0 IME1 matrix kernel for SpacemiT X60 On the SpacemiT X60, IME matrix acceleration only covered Q4_0/Q4_1/Q4_K. Q8_0 had no IME1 kernel, and since the SpacemiT build sets GGML_CPU_REPACK=OFF there was no repack path compiled in either, so Q8_0 had no accelerated path at all and ran roughly ten times slower than Q4_0 for prefill on the same board. - add make_block_q8_0x16 and the Q8_0 repack entry: interleave the weights into the 16-column layout the IME1 vmadot sequence expects - add ime1::gemm_kernel_i8i8, an int8 x int8 IME1 kernel with a single-row and a 4-row A path; the 4-row path loads each B panel once and reuses it across 4 rows of A - add quantize_a_4row_i8 for the 4-row activation quantization - wire both into forward_mul_mat and the repack factory for Q8_0 - docs: mark Q8_0 as supported on X60 Correctness was checked against a quant-exact integer reference for K = 32 up to 4096, with a max relative error of about 1e-6, and by checking that generation stays coherent across several prompts. Tested on Milk-V Jupiter (SpacemiT X60), Bianbu 2.1.1, gcc 14.2, with Qwen2.5-0.5B-Instruct Q8_0. llama-bench -t 4 under taskset -c 0-3, 5 repetitions on an idle board: pp128 goes from 10.70 to 93.87 t/s. Q4_0 is unchanged at 106.40 -> 107.51 t/s, as expected since this does not touch that path. * ggml-cpu : move q8_0_16x32 decl to IME1 section * ggml-cpu : align q8_0 IME1 kernel assignments
…rotation metadata (ggml-org#28498) * kv-cache: save exact KV rotation metadata, reject restoring mismatched rotation * tests : move the state rotation test to test-save-load-state the test is now part of the save/load test matrix and runs against every model under test, like the rest of the suite it probes the KV cache type combinations supported by the model and treats models that do not use attention rotation as passing vacuously Assisted-by: pi:llama.cpp/Qwen3.8-27B * cont : skip unsupported KV caches --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
…org#29958) * llama : fix unexpected graph reallocation in the k-pool models Both k-pool models built a graph shape that depends on state the full-context reserve cannot know: - qwen4exp branched on inp->cache_safe, which turns false as soon as llama_memory_seq_cp shares cells (e.g. batched-bench -pps): the QSA layers swapped scatter+gather for fill+concat and dropped the new_pool_rep leaf, so the decode graph had 12 fewer nodes than the reserved one - glm5-next branched on gather = n_tokens <= 16 && n_kv > n_sel, so the TG decode built the gather shape (7564 nodes) while the last reserve, the PP one, had the dense shape (7762 nodes) Either mismatch forces a decode-time re-reserve that drops the worst-case sizing and bakes in the current state, so the next state growth (n_pool, n_kv, n_new) needs more room at an unchanged graph size and aborts under GGML_SCHED_DEBUG_REALLOC=1. Reproduce with, e.g.: GGML_SCHED_DEBUG_REALLOC=1 ./bin/llama-batched-bench \ -hf ggml-org/GLM-5.3-Flash-GGUF:Q2_K -npp 2500 -ntg 32 -npl 1,2 \ -c 32768 -pps -kvu Always scatter+gather the pooled keys, and pick gather from context constants only: n_ubatch bounds every ubatch, top_k + kpool - 1 bounds n_sel. Every graph of a context then shares one shape, which the reserve covers, and the dense path measured faster than the gather path at 2.5k and 16k context. Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD * llama : drop the unused k-pool cache_safe graph API The k-pool graphs no longer branch on cache_safe, so nothing reads get_kpool_cache_safe() or the conditional new_pool_rep any more: both models always pass the scatter target, which set_input_kpool now requires instead of merely preferring. Also drop the cache_safe copy in kpool_build_sizes(), a sizes-only helper. The layout and state flag itself stays, it still decides which pools a layout with shared cells must re-pool. Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD * tests : add a shared-seq graph reserve regression test Decode a prompt into seq 0, share its cells with seq 1 via llama_memory_seq_cp (what llama-batched-bench does for -pps), then keep decoding both sequences. For the k-pool models sharing clears cache_safe, which changes the graph topology while the pools keep growing, so a scheduler that re-reserves with the current state instead of the worst-case one aborts under GGML_SCHED_DEBUG_REALLOC=1. The test registration sets that flag, and the test aborts on both k-pool models before 2220411. kimi-linear and minimax-01 are skipped: they reserve the final pp graph with n_seqs = 1 (see [TAG_RESERVE_DIAG_DECAY] in llama-context.cpp), so every multi-seq graph has a different layout and re-reserves by design. Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-MOPD * cont : add TODOs * cont : fix comment * cuda: match the moe weighted reduction on empty ubatches ggml_cuda_match_moe_weighted_reduction rejected tensors with zero rows. A ubatch without outputs shrinks the last layer to zero rows through inp_out_ids, so graph_optimize dropped its alloc dep there and the scheduler graph lost one node compared to the reserved one. The scheduler then re-reserved at the size of that ubatch, and the next ubatch with the same node count but larger tensors aborted under GGML_SCHED_DEBUG_REALLOC=1. The compute loop already skips empty nodes before trying any fusion, so the guard only made the alloc deps depend on the row count. * tests: build the rollback test only where internal symbols link The shared-seq case calls llm_arch_from_string, which libllama does not export through LLAMA_API, so linking test-recurrent-state-rollback fails on Windows with shared libraries. Its build now sits in the NOT WIN32 OR NOT BUILD_SHARED_LIBS block, next to test-llama-archs and the test registration it already lives under. * tests: skip archs by name in the shared-seq reserve test The skip of kimi-linear and minimax-01 went through llm_arch_from_string, which libllama does not export through LLAMA_API, so the test could not link on Windows with shared libraries. It now compares the general.architecture string directly, and the test builds on every platform again. --------- Co-authored-by: Pascal <admin@serveurperso.com>
* vulkan: sparse flash attention for quantized K/V Assisted-by: Claude * vulkan: single-scan sparse FA index compaction The compaction ran one workgroup per mask row and walked the row in BLOCK_SIZE chunks, with a workgroup scan per chunk. For decode that is one workgroup doing KV/1024 barrier-bound iterations, so at 128k cells it cost more than the sparse attention it feeds. Split the row into contiguous segments instead: one per subgroup with ballot counting over coalesced loads, or one per thread without subgroups. A single scan over the segment counts then gives each segment its output offset. The index list stays ascending.
* server: reject partial media truncation * server: keep only the keep_first fix Drop the mtmd test helper change, which no longer builds since clip_image_f32_batch stores its entries by value, and drop the vision test: no test fixture reaches a cut between two adjacent media chunks with a reused cache (tinygemma3 uses SWA and wraps images in text tokens, tinyopenjev and small-test are recurrent), so the test passed or failed independently of the fix. --------- Co-authored-by: Pascal <admin@serveurperso.com>
* ggml_cuda: optimize accumulation in mmq_vec_dot_fp4_fp4_mma for better performance * remove whitespace * fix: correct indentation in mma_block_scaled_fp4 loop
* server: support vision input for Clef * move input_attn_causal to private * extend old server_batch::embd * server_batch::token::pos to multi dim * nits * fix abort * fix img tokens cap * fix yield_to_queue mutate data
…tatistics (ggml-org#27990) * webui: Use toLocaleString() format consistently across chat message statistics * Add missing semicolon * ui: respect lint
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* cuda: stage the lightning indexer queries in head passes for MUSA MUSA archs 21 and 22 cap static shared memory at 28 KB, and the tile kernel staged the queries of all four heads next to the key tile for 33 KB. The queries are now staged in passes of LIGHTNING_INDEXER_TILE_HEADS_PER_PASS heads: two on MUSA for 25 KB, four elsewhere where the single pass folds to the previous kernel. * cuda: use the vector lightning indexer kernel on MUSA Address review from am17an: the tile kernel stays off MUSA, whose archs 21 and 22 cap static shared memory at 28 KB, below the 33 KB the tile needs, so MUSA keeps the vector kernel it ran before. This replaces the head passes, CUDA and ROCm run the merged kernel unchanged.
…29936) * Fix Intel prefill regression on MoE models * Revert the n_per_expert change back to nei1
tojson printed floats through a C++ stream (6 significant digits, no ".0"), so 48.858844 became 48.8588 and 1.0 became 1 in tool definitions, tool-call arguments and other template output. Write the shortest digits that read back as the same value, as json.dumps does. The string and safe filters of a float use the same code.
) * cuda : use SIMT implicit GEMM conv2d on AMD without WMMA Assisted-by: Claude Opus * cuda : update conv2d SIMT comment * cuda : skip SIMT conv2d on AMD for short reductions Co-authored-by: thelittlefireman <5165783+thelittlefireman@users.noreply.github.com> --------- Co-authored-by: thelittlefireman <5165783+thelittlefireman@users.noreply.github.com>
- add rpc_log_format_size() and print byte counts in KiB/MiB/GiB across the
RPC client, server and transport traces
- add rpc_log_format_tensor() and include tensor name, op, type and shape in
the set/get/memset/copy/init tensor logs
- fix rpc_cmd_name() returning UNKNOWN for SET_TENSOR_2D, GET_TENSOR_2D,
COMM_INIT, COMM_ALLREDUCE and COMM_FREE
- move the log helpers to ggml-rpc/log.{h,cpp} under a common rpc_log_ prefix
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
…rg#29677) When every device reports zero free memory the split points are normalized by a zero sum, the NaN split sends layers to a device index past the end and loading aborts with vector::_M_range_check.
…gml-org#30407) Added that the oneapi toolkit includes the sycl dependencies. This makes it clear for first time users, that the toolkit includes all necessary dependencies, so that the user is guided to install the toolkit if they do not already have the dependencies separately
* common : add --spec-draft-n-ubatch Allow configuring the physical batch size of the draft context. The drafters feed it at most the drafted tokens per sequence and chunk larger batches by its own n_ubatch, so it does not need the ubatch of the target. A smaller value shrinks the draft compute buffer and leaves more VRAM for the target KV cache (the draft is fitted together with the target). The default 0 keeps the current behavior of following --n-ubatch. Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next * cont : remove comment [no ci]
* ggml: fix conv_1d result layout for batched input The GEMM behind conv_1d folds the batch into its rows, so the product sits in memory as [OL, N, OC] while the reshape claims [OL, OC, N]. The shape is right and the contents are transposed, so nothing fires: no assert, no abort, the graph just runs and the channels come out interleaved with the batches. With N == 1 the two readings describe the same buffer, which is why the three vocoders in the tree never hit it, they all squeeze to 2d right after the call. Reshape to the real layout and permute OC and N back. The N == 1 path keeps its direct reshape, so the current callers build the same graph and pay for no extra copy. tests/test-conv-1d.cpp compares one batched convolution against N single batch convolutions built from the same weight and the same input slices. Five of the seven geometries fail before this change and all seven pass after. N == 1 and OC == 1 pass either way, since swapping two dimensions when one of them is 1 changes nothing. The batched path matters to downstream audio projects where several synthesis lanes share a single graph. Nothing in mtmd batches a vocoder today, each one streams one flow and carries its left context between calls, but the op is correct now if one of them grows a lane dimension. * ggml: build the same conv_1d graph for every batch size Address review from ggerganov: drop the N == 1 branch of ggml_conv_1d. The GEMM result is always reshaped to [OC, N, OL] and permuted back, as ggml_conv_2d does, so every batch size builds the same graph. For N == 1 this adds a cont on an already ordered tensor.
* skill : add tools/ui review checklist to code-review skill Adds a WebUI (tools/ui) section distilled from a year of review feedback on tools/ui PRs: constants/enums placement, layering (stores/services/utils/hooks), DRY and dead code, Svelte 5 idioms, comment hygiene, streaming perf, frontend security, and PR hygiene (static build, demos, scope discipline). Scoped to tools/ui only. * skill : expand tools/ui constants/enums ruleset Replaces the short constants bullet list with a decision ruleset distilled from review outcomes: when a literal becomes a string enum vs an enum-keyed map vs a declarative config vs a grouped as-const object vs a scalar constant, naming and suffix conventions (_REGEX, _MS, _PX, semantic separators), placement and composition rules, and what legitimately stays inline. * skill : drop stale static-build-commit requirement for tools/ui Static build output is no longer stored in the repo - the UI is built by CI and bundled at release time. Keep the requirement that npm run build succeeds and the UI build job is green. * skill : address copilot review of tools/ui checklist - scope the video-demo requirement to user-facing interactive/visual changes instead of every tools/ui PR - narrow the constants trigger to reused/domain-significant literals, keeping the inline boundary as the escape hatch - carve out independent literals for persisted/wire-contract test assertions so they don't become tautological via shared constants - allow simple presentation-local $derived in components; only shared business logic, orchestration and parsing leave the component - match the existing barrel documentation style instead of requiring per-export JSDoc everywhere - keep non-obvious why/rationale comments; move only examples and edge-case enumerations into tests - point at the actual interpolate-size/grid-template-rows transition pattern in CollapsibleRegion.svelte instead of calc-size() * skill : use spaces for nested list indentation EditorConfig CI requires space indentation for all files; the scalar-constants sub-bullets used tabs. Matches the 2-space nested list convention used in docs/.
* cuda, hip: GDN chunked kernel Ported-from: tetherto#257 Source-commit: 53f0027 Co-authored-by: Aman Gupta <aman.gupta@tether.io> * fix older builds * review fixes * cuda: enable GDN MMA on tested Ampere and Ada GPUs Enable SM86 and SM89 after RTX 3090 and RTX 4090 correctness and prefill benchmarks. Keep other untested architectures on the existing fallback. Assisted-by: Codex * cuda : use refactored swizzle API in GDN MMA * ggml : GDN only guarantees the final state when n_tokens > K * cuda : add short GDN MMA phase comments --------- Co-authored-by: gianni-cor <gianfranco.cordella@tether.io> Co-authored-by: Aman Gupta <aman.gupta@tether.io>
…30454) * ggml-cpu : size set_rows conversion buffer for all threads SET_ROWS is planned with n_tasks = 1, but every thread runs it and uses its own row of the F16 -> F32 conversion buffer, so threads > 0 wrote past the work buffer when the graph had another multi-threaded op. * move set_rows to n_task = n_threads group
* docs: update zdnn lib install + add notes on running models Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * docs: change title Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * docs: add note about minimum version Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> --------- Signed-off-by: Aaron Teo <aaron.teo1@ibm.com>
…g#30342) (ggml-org#30343) * ggml-cpu : keep the FP16->FP32 lookup table on x86 with F16C (ggml-org#30342) ggml-org#30270 routed both scalar conversions through F16C. For FP16->FP32, Clang 20.1.8 lowers the intrinsic to a vpinsrw merging into the live accumulator register of the quantized dot products, which serialises the loop (19 vs 4 cycles/iteration estimated, 45-65% slower Q8_0/Q4_0 MUL_MAT_ID kernels measured). Keep F16C for FP32->FP16 as intended and the table for FP16->FP32. * ggml-cpu : comment out the F16C FP16->FP32 define instead of removing it (review)
* sycl: persistent IQ3_S/IQ3_XXS weight reorder for SYCL decode and prefill Add persistent reordered weight layouts for IQ3_S and IQ3_XXS. Weights are reordered once at load time; the reordered MMVQ dot products then walk dense super-blocks with dp4a and SWAR sign expansion instead of gathering bits from the interleaved standard layout. Reorder-aware dequantization feeds the batched prefill GEMMs so prompt processing sees the same benefit. Also harden the oneDNN build check: a missing DNNL package or a CPU-only DNNL build now fails at configure time instead of silently producing a degraded backend. Measured on Intel Arc Pro B70 with Qwen3.8-27B-GSQ-RCO-IQ3_S (MTP), -ngl 999 -b 4096 -ub 1024 -fa on: tg256 @ 0 ctx 12.92 -> 23.54 tok/s (+82%) tg256 @ 9,216 12.33 -> 21.69 tok/s (+76%) tg256 @ 64,000 9.76 -> 14.91 tok/s (+53%) pp8192 833.9 -> 833.6 tok/s (flat) Qwen3.5-9B Q4_K_M / Q3_K_M are unchanged within noise (largest delta -0.5% pp, tg flat), so standard quants are not affected. The MMVQ GLU fusion widening and rms_norm+scale fusion are intentionally not part of this change; master already carries them via ggml-org#28931. * removing wait as per review request * sycl : fix fusion.cpp line ending
The clamp kernel indexed source and destination as one flat contiguous array, so a strided view as input (the new CLAMP v=1 case) read the wrong elements. Add a strided kernel_clamp_nc, as the other unary ops have, and take it whenever the source is not contiguous.
…-upstream-1014 # Conflicts: # README.md
…-1014 Jimwu.master sync upstream 1014
Conflict resolution: - mmq: adopt upstream prec_src1 template param; fork RDNA3.5/ROCMFP4 kernels use GGML_PREC_Q8. Upstream moved J selection to host (mmq.cu), so the fork's RDNA3.5 J tuning moves to mmq_rdna35_select_J(). ggml_cuda_mmq_needs_fallback() takes cc (RDNA3.5 uses I == 64). Fork compacted-MoE grid kept. - gated_delta_net: keep fork RDNA3.5 chunked kernel first, then upstream MMA GDN path. - ggml-cuda supports_op: keep ROCMFP4 F16-activation accept, add upstream fwht and MUL_MAT_ID PREC_F32 checks. - metal: nullable src[3] + fork SSM_CONV layout guard. - test-backend-ops: take upstream tests. Co-Authored-By: Claude <noreply@anthropic.com>
Upstream ggml-org#29184 fuses the MoE shared expert into MMVQ via fusion->shared_up/shared_gate/shared_dst. mmvdq ignored those fields, so on RDNA3.5 the fused op left shared_dst unwritten (MUL_MAT_ID_SHARED n=1 failed). Fall back to mmvq when a shared expert is fused. Co-Authored-By: Claude <noreply@anthropic.com>
llama.cpp PR #151 versus gfx11: full regression catalogPR #151 has reproducible performance regressions. The largest are prompt-dependent MTP throughput losses of about 3%, 5%, and 12%. A controlled one-line GDN diagnostic identifies the main cause. Smaller ordinary decode and prefill losses involve normalization overhead and unnecessary transfers for mixed-input graph branches. All original benchmark status and sanity checks passed. Tested revisions and coverage
The first-pass median TTFT change is -0.40% for LLMs and -1.38% for VLMs. Median decode TPS changes are -0.58% and -0.42%, respectively. Decode is lower in 49/50 LLM rows and 44/50 VLM rows, so the common small loss should not be dismissed merely because many rows are below the 2% screening threshold. Main repeated regressionsValues below come from the two independent repeat pairs, excluding the initial screening run. Negative TPS change means slower generation; positive TTFT change means slower first token.
Smaller repeated effects remain: Qwen3-1.7B/4B pp128 TTFT is +1.67%/+1.74%; Cosmos Q8 VLM TTFT is +1.73%; Qwen2.5-0.5B pp128 decode is -1.88%; Gemma-4-E2B Q4_K_M/Q4_0 decode is -1.39%/-1.51%. The original Gemma-3-4B pp128 TTFT candidate did not reproduce (-0.81% on repeats). The original Qwen3.6-27B MTP code prompt-time candidate also did not reproduce (-0.78%). The PR author's Q8_0 pp128 concern was checked with Cosmos Q8: repeated TTFT is +1.07%, rather than a confirmed 2.5% loss in this setup. Cause 1: changed GDN reduction order changes MTP output and acceptanceUpstream commit For the affected prompts, both ordinary generation and MTP produce different continuations. The MTP loop then produces fewer useful output tokens per verification cycle. Server timing decomposition shows:
The verification cycle is slightly faster. Lower output yield per cycle accounts for the throughput loss. Cycle time includes all server evaluation work and must not be interpreted as every individual kernel becoming faster. Acceptance percentage alone is also insufficient: some list prompts change the number of proposed tokens without changing useful tokens per verification step. Controlled confirmation: an isolated PR library changes only The separate normalization diagnostic leaves the PR answers unchanged. Exact This is a real request-level throughput change, even though it comes from different continuations and draft acceptance rather than slower verification. It is not an n_max=1 identity failure or evidence by itself of lower model accuracy. For 2B code, gfx11 completes at 187 tokens; the PR's longer answer reaches the 200-token budget mid-code. Evidence: probe analysis, GDN diagnostic diff, repeated verification-step analysis, input parity. Cause 2: normalization's large-grid fix adds overhead to ordinary shapesUpstream An isolated diagnostic removes only these loops and trailing barriers while retaining the rest of the PR. Two runs of 20 repetitions per variant, ordered gfx11 / PR / diagnostic / diagnostic / PR / gfx11, show:
Thus normalization is a demonstrated contributor, but it does not explain the whole ordinary-model regression. Stock Qwen3 traces also show normalization as the largest growing kernel family with unchanged normalization counts. The diagnostic is not a production patch: a proper fix should keep the large-grid path and provide an efficient ordinary-size path. Evidence: ablation timings and samples, compiled-kernel comparison, diagnostic diff. Cause 3: mixed-input graph support adds transfers during token-only decodeCommit
These transfers introduce host API overhead as well as small GPU copy launches. The copy-instrumented CPU language-call scope increases by about 36 microseconds for Qwen and 32 microseconds for Gemma. Those are diagnostic scope differences, not isolated causal savings or end-to-end token latency: the calls are asynchronous, and tracing affects timing. The tensor names and source branch establish where the extra work comes from. A production optimization should avoid preparing inactive branch inputs while preserving later mixed-input requests and graph reuse. Globally disabling mixed-input support would discard the capability added by the upstream change. No production fix was made in this investigation. Evidence: Qwen gfx11 copy attribution, Qwen PR copy attribution, Gemma PR copy attribution, source change. Smaller VLM effects and limits of attributionCosmos Q4 and Q8 VLM process the same 1322 tokens in every selected profiled request. Kernel counts and language/vision work sizes match. The loss is distributed, rather than one attention explosion. Median request kernel totals show language normalization growing by about 6.6/6.3 ms and vision normalization by 1.8/1.6 ms. Vision attention grows by about 7.5/3.7 ms, while vision conversion improves by about 7.7/6.5 ms. Language matmul grows modestly, about 10.7/5.0 ms. These profiled deltas do not fully account for the unprofiled +33.3/+24.7 ms TTFT changes; host work and tracing variability limit exact additive attribution. The remaining matmul/attention differences have not been isolated to individual commits. Gemma-4-E4B text-short-smoke has 21 input and six output tokens. Its repeat medians give 178.43 to 182.51 ms total response time, about 4.1 ms more, but the two PR runs differ considerably. Reversed profile collection confirms an approximately 0.6 ms increase in normalization GPU work across the six-token decode, plus additional copy work. The remaining client/server overhead is not fully isolated. Treat -3.83% as a variable short-response measurement, not a general E4B generation-rate regression of that size. Some VLM traces contain isolated 100-400 ms stalls. They appear in both builds and in different kernel families on reversed runs. All events are retained. The VLM discussion uses per-request medians, and the unprofiled repeats remain the primary performance evidence; trace averages containing these stalls must not replace them. Longer generated answers are separately recorded. For example, Qwen3.6-35B large-context VLM has the same 16262 input tokens: first-pass TTFT improves 10331.28 to 10262.36 ms and TPS improves 55.95 to 56.10, but output grows from 206 to 294 tokens, making total response time 14.07 to 15.56 seconds. That extra response time is explained by generating more text. AssessmentThe full catalog passes its existing sanity checks, but the PR should not be described as performance-neutral. The largest MTP losses have a demonstrated numerical-order cause. The common smaller decode loss has both a demonstrated normalization contributor and traced unnecessary mixed-input transfers. Small residual VLM and host costs remain only partially attributed. These measurements do not justify discarding all lower results as noise or as changed input length. The most concrete follow-ups are an ordinary-size normalization fast path, avoiding inactive mixed-input transfers, and evaluating the GDN schedule with MTP acceptance and output behavior as well as kernel speed. The isolated diagnostics are evidence for those directions, not validated production changes. The full first pass and every independent repeat are included below. CSV contains primary metrics, repeat medians, and findings; repeat JSON retains individual run values; final audit records status, input parity, checkpoints, and unchanged source/binary hashes. Original raw logs, commands, configs, and traces remain in this directory. Independent repeatsTwo fresh runs per build, with reversed arm order. Each regular run retains 10 catalog repetitions; MTP repeats retain five repetitions per prompt. These estimates exclude the first pass. Positive TTFT change means slower; negative TPS change means slower.
Full regression catalog: first passAll 166 rows below passed the original catalog sanity checks. Input and output lengths are actual measured token counts. Workload changes include generated length and saved MTP output text changes; they are not silently excluded from the comparison. The CSV contains all raw metrics, hashes, repeated metrics, and findings. LLM TTFT is prefill-derived, VLM TTFT is streaming latency, and MTP TTFT is server prompt time.
|
Annieren
approved these changes
Oct 19, 2026
Annieren
left a comment
Annieren
left a comment
There was a problem hiding this comment.
I will work on MTP improvement on RDNA3.5.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge origin/master into gfx11.
Conflict resolution:
Fix: mmvdq now skips fused shared experts (upstream ggml-org#29184); MUL_MAT_ID_SHARED n=1 failed on RDNA3.5.
Testing (gfx1151):