fix(patches): 0022 quantizes an expert's rows once; measured with real weights

.github/workflows/publish.yml

@@ -2702,10 +2702,13 @@ jobs:
           echo "CXX=g++-14" >> "$GITHUB_ENV"
       - name: Display CPU Info
         uses: bernardladenthin/actions/common/print-host-info@44390bd464395f560f4170d2c4bfb7a6348541b3 # v1.1.0
+      # armv8.2-a+dotprod: the plain ARMv8 baseline has no repacked CPU weights at all, so all 24
+      # test_moe_repack cases (llama/patches/0022) skipped here; dotprod selects the q4_0_4x4 and
+      # q4_K_8x4 kernels, the ones an N1-class server runs (the shipped jar's armv8.2 modules)
       - name: Build libraries
         run: |
           mvn -q --no-transfer-progress -f llama/pom.xml compile
-          .github/build.sh "-DOS_NAME=Linux -DOS_ARCH=aarch64 -DGGML_NATIVE=OFF -DBUILD_TESTING=ON"
+          .github/build.sh "-DOS_NAME=Linux -DOS_ARCH=aarch64 -DGGML_NATIVE=OFF -DGGML_CPU_ARM_ARCH=armv8.2-a+dotprod+fp16 -DBUILD_TESTING=ON"
       - name: Verify llama.cpp patches are applied
         run: .github/verify-patches-applied.sh
       - name: Run C++ unit tests

CHANGELOG.md

@@ -66,6 +66,14 @@ from version 5.0.0 onward. Pre-fork releases (`1.x`–`4.2.0`) were authored by
   An app that used checker-qual's annotations itself through the AAR has to declare it.
 
 ### Added
+- **Faster prompt processing for MoE models on the CPU** (`llama/patches/0022`): the expert matmuls
+  over repacked weights batch the tokens of an expert through `gemm` and hand their work out
+  dynamically, instead of one gemv per token on a fixed thread split. With trained weights
+  (gpt-oss-20b MXFP4, Qwen3-30B-A3B Q4_K_M, OLMoE-1B-7B Q4_0) prompt processing is +13 % to +66 %
+  on a Zen 3, a Neoverse-N1 and an IBM z16, generation unchanged
+  ([`docs/benchmarks/moe-repack-real-weights.md`](docs/benchmarks/moe-repack-real-weights.md)).
+  `LlamaModel.setCpuMoeBatched(false)` (or `GGML_CPU_REPACK_MMID_BATCHED=0`) restores upstream's
+  path for a comparison on one's own machine.
 - **`TextToSpeech.supportsSpeakerReference()`** (and `FfmTextToSpeech.supportsSpeakerReference()`,
   C API `jllama_tts_supports_speaker_reference()`): whether the loaded pair clones a voice from a
   speaker reference clip, llama.cpp's `mtmd_helper_cap::audio_gen_spk_ref` (b11901, #31183) --

README.md

@@ -1688,6 +1688,9 @@ core takes fewer of them. Upstream llama.cpp multiplies one token at a time and
 fixed share. The results are the same up to rounding; measured on gpt-oss-20b in a sandbox, prompt
 processing was 47 % faster and token generation 26 %
 ([`docs/benchmarks/moe-mxfp4-cpu-alderlake.md`](docs/benchmarks/moe-mxfp4-cpu-alderlake.md)).
+With trained weights (gpt-oss-20b, Qwen3-30B-A3B, OLMoE) prompt processing was 13 % to 66 % faster
+on a Zen 3, a Neoverse-N1 and an IBM z16, token generation unchanged
+([`docs/benchmarks/moe-repack-real-weights.md`](docs/benchmarks/moe-repack-real-weights.md)).
 
 Both paths stay available, so the difference can be measured on your own machine:
 

TODO.md

@@ -165,7 +165,7 @@ Kept as a list because each entry names the machine it needs -- the harness is
 | **OpenVINO on an Intel GPU/NPU** | suitable Intel device, unavailable to these sessions | the 21.7x/336x figures are CPU-plugin fallback only; they do not prove GPU/NPU execution |
 | the two **SME** aarch64 modules | Linux aarch64 with SME, absent from current CI/cloud machines | macOS's static Metal library does not exercise these modules |
 | **gpt-oss-20b on both MoE paths of `llama/patches/0022`**, on the **Core Ultra 7 165U** (`-t 6/8/10`) | the laptop; switch with `LlamaModel.setCpuMoeBatched` or `GGML_CPU_REPACK_MMID_BATCHED=0` ([`docs/benchmarks/moe-mxfp4-cpu-alderlake.md`](docs/benchmarks/moe-mxfp4-cpu-alderlake.md)) | upstream's static work split meets real P-, E- and LP-E-cores only there; the sandbox could emulate one slow core for prompt processing (−49 % upstream's path, −24 % batched) but not for generation |
-| **`llama/patches/0022` with real weights on every CPU family whose repack path it changes**: gpt-oss-20b (MXFP4) and a Q4_K MoE (e.g. Qwen3-30B-A3B Q4_K_M), both switch positions -- `llama-perplexity --kl-divergence` of the batched path against upstream's, `llama-bench` pp/tg at several thread counts, and once through the jar (`LlamaModel.setCpuMoeBatched`) | x86-64 with AVX-512/AMX and Zen 4 (EC2 or Vultr: the `sapphirerapids` and `zen4` modules); aarch64 with dotprod (cfarm424, Neoverse-N1, 64 cores) and with i8mm/SVE (Graviton 3/4); s390x with VXE (cfarm191, z16 -- ggml repacks only Q4_0 there, so a Q4_0 MoE converted to big-endian, on the vector modules `z15`/`z16`, see "Linux s390x" in CLAUDE.md) | #578 measured random weights in the real shapes, on the `alderlake` module of an AVX-512 host. KLD 0 there shows the batched path is right for that gemm kernel, not for the NEON, i8mm or VXE kernels it calls elsewhere, and random weights route tokens to experts more evenly than a trained model does |
+| **`llama/patches/0022` with real weights on the CPU families still missing** -- done on Zen 3 (AVX2, `haswell`), a Neoverse-N1 (dotprod, `armv8.2_2`) and an IBM z16 (VXE, `z16`), with gpt-oss-20b, Qwen3-30B-A3B Q4_K_M and OLMoE Q4_0 ([`docs/benchmarks/moe-repack-real-weights.md`](docs/benchmarks/moe-repack-real-weights.md): +13 % to +66 % prompt processing, generation unchanged, KLD 0 for MXFP4/Q4_0); the same `llama-bench` + `llama-perplexity --kl-divergence` pair, both switch positions, on the rest | x86-64 with AVX-512/AMX and Zen 4 (EC2 or Vultr, paid: the `sapphirerapids` and `zen4` modules); aarch64 with i8mm/SVE (Graviton 3/4, paid: the `q4_K_8x8` / `q4_0_4x8` kernels) | each module calls other gemm kernels; the N1 run found a defect (Q4_K −18 % at 32 threads, fixed) that x86 did not show. The C++ guard (`test_moe_repack`) still skips on Windows arm64 and s390x, where nothing is repacked in the test build |
 | [`docs/local-test-plan.md`](docs/local-test-plan.md) parts A–D on the **Core Ultra 7 165U** | the laptop the Windows guide (#575) is written for | part A -- which CPU module ggml picks there and what it gains -- is what CI cannot measure; the only report so far is the Zen 3 one (`docs/handover/local-agent-report-b11538-windows.md`). One result file per machine, unmeasured values marked as such |
 
 ## Open — jllama-specific

docs/benchmarks/README.md

@@ -15,6 +15,7 @@ themselves live in [`../../TODO.md`](../../TODO.md).
 |---|---|
 | [`cpu-backend-modules.md`](cpu-backend-modules.md) | What each of the 22 shipped CPU backend modules is worth, per microarchitecture — and whether `ggml_backend_load_best` picks the fastest one |
 | [`moe-mxfp4-cpu-alderlake.md`](moe-mxfp4-cpu-alderlake.md) | Where the `alderlake` module (Core Ultra) loses time on Qwen3 and gpt-oss-20b, measured with random-weight models in the real shapes -- and what the fix to ggml's MoE path in `llama/patches/0022` is worth (+47 % prompt, +26 % generation; switchable for A/B) |
+| [`moe-repack-real-weights.md`](moe-repack-real-weights.md) | `llama/patches/0022` with trained weights (gpt-oss-20b MXFP4, Qwen3-30B-A3B Q4_K_M, OLMoE Q4_0) on Zen 3, a Neoverse-N1 and an IBM z16: +13 % to +66 % prompt processing, generation unchanged, KLD against upstream's path -- and the ARM defect of the first version it found |
 | [`gpu-module-jars.md`](gpu-module-jars.md) | What happens when a GPU module jar is actually *used* -- two findings that contradict the "a module that cannot help is skipped" premise |
 | [`arm-fp16-conversion.md`](arm-fp16-conversion.md) | Every shipped ARM binary converts fp32 to fp16 in software, ours and upstream's: 26 of 26 disassembled, 8.9x to 13.3x on the function, and 0.03 % of cycles so inference does not move |
 | [`upstream-f16c-30270.md`](upstream-f16c-30270.md) | An isolation of one upstream commit: what it does, that it changes nothing measurable here, and a compile-time guard proven red against the defect |

docs/benchmarks/moe-repack-real-weights.md

@@ -0,0 +1,167 @@
+<!--
+SPDX-FileCopyrightText: 2026 Bernard Ladenthin <bernard.ladenthin@gmail.com>
+
+SPDX-License-Identifier: MIT OR Apache-2.0
+-->
+
+# Patch 0022 with real weights
+
+**The question.** [`llama/patches/0022`](../patches.md#0022-ggml-cpu-repack-batch-the-tokens-of-an-expertpatch)
+batches the tokens of an expert in the CPU backend's repacked `MUL_MAT_ID` and hands its work out
+dynamically. It was measured on random weights in gpt-oss-20b's shape, on one x86 module
+([`moe-mxfp4-cpu-alderlake.md`](moe-mxfp4-cpu-alderlake.md)). Does it hold with trained weights,
+whose routers spread tokens unevenly, on the other kernels it calls: ARM's, K-quants?
+
+**The answer, measured 2026-10-27 on b11901 + `llama/patches/`:**
+
+1. **Prompt processing gains on every model and machine measured**: +13 % to +66 %, on x86
+   (Zen 3), ARM (Neoverse-N1) and s390x (z16).
+2. **Token generation is unchanged** within noise. With one token per expert, nothing is batched.
+3. **The logits are those of upstream's path for MXFP4 and Q4_0** (KLD 0.000000, same top token
+   100 %). **For Q4_K they differ by a KLD of 0.0075**, which is not a defect of the patch:
+   upstream's own repack path is as far from the unrepacked computation (below).
+4. **It found a defect in the first version of the patch.** On a Neoverse-N1, Q4_K prompt
+   processing was 18 % slower than upstream's path at 32 threads, because every column chunk
+   quantized the expert's activations again. ARM has no SIMD form of the Q8_K quantizer. Fixed:
+   each group of four rows is now quantized once.
+5. **The ARM C++ tests never ran the patch's guard.** `test-cpp-linux-aarch64` built for plain
+   ARMv8, where nothing is repacked, so all 24 `test_moe_repack` cases skipped. It builds with
+   dotprod now; on the N1 all 25 tests pass.
+
+## Setup
+
+- **Models** (trained weights from Hugging Face):
+  - gpt-oss-20b MXFP4 (`ggml-org/gpt-oss-20b-GGUF`, 12.1 GB);
+  - Qwen3-30B-A3B Q4_K_M (`unsloth/Qwen3-30B-A3B-GGUF`, 18.6 GB; 128 experts, 8 used);
+  - OLMoE-1B-7B-0125-Instruct Q4_0 (`allenai/OLMoE-1B-7B-0125-Instruct-GGUF`, 3.9 GB; 64 experts,
+    8 used).
+- **The build:** llama.cpp b11901 with the 14 patches, built as the modular library jars are
+  (`GGML_BACKEND_DL`, every CPU variant, no OpenMP, clang). ggml picks the module, upstream's
+  `llama-bench` and `llama-perplexity` measure.
+- **The switch:** `GGML_CPU_REPACK_MMID_BATCHED=0` gives upstream's path, `1` the batched one, in
+  the same binary.
+- **Benchmarks:** `llama-bench -p 512 -n 128 -r 3`, off and on interleaved per thread count.
+- **KLD:** `llama-perplexity --kl-divergence` on wikitext-2 (20 chunks of 512), the batched path
+  against upstream's.
+- **The machines:**
+  - **x86:** Ryzen 7 5800H (Zen 3, 8 cores / 16 threads, module `haswell`), Docker on Windows;
+    downloads were running during part of the first series.
+  - **ARM:** Neoverse-N1, cfarm424 (64 cores, dotprod, no i8mm, module `armv8.2_2`), a shared
+    host; some rounds ran next to other users' load.
+  - **s390x:** IBM z16, cfarm191 (8 CPUs, module `z16`), OLMoE only ([below](#s390x)).
+
+## Prompt processing (pp512, t/s, off → on)
+
+| model (type) | machine | threads | upstream | batched (final) | |
+|---|---|--:|--:|--:|--:|
+| gpt-oss-20b (MXFP4) | Zen 3 | 4 | 28.56 | 37.23 | +30 % |
+|  |  | 8 | 41.68 | 59.58 | +43 % |
+|  |  | 16 (SMT) | 54.44 | 69.80 | +28 % |
+|  | N1 | 8 | 49.32 | 80.85 | +64 % |
+|  |  | 16 | 98.28 | 152.46 | +55 % |
+|  |  | 32 | 176.97 | 278.70 | +57 % |
+| Qwen3-30B-A3B (Q4_K) | Zen 3 | 4 | 39.96 | 53.27 | +33 % |
+|  |  | 8 | 58.32 | 84.65 | +45 % |
+|  |  | 16 (SMT) | 71.72 | 82.27 | +15 % |
+|  | N1 | 8 | 56.81 | 66.77 | +18 % |
+|  |  | 16 | 105.99 | 121.84 | +15 % |
+|  |  | 32 | 190.66 | 215.69 | +13 % |
+| OLMoE-1B-7B (Q4_0) | Zen 3 | 4 | 81.11 | 116.78 | +44 % |
+|  |  | 8 | 138.81 | 189.95 | +37 % |
+|  |  | 16 (SMT) | 204.83 | 236.12 | +15 % |
+|  | N1 | 8 | 192.28 | 274.53 | +43 % |
+|  |  | 16 | 378.50 | 506.39 | +34 % |
+|  |  | 32 | 705.85 | 989.98 | +40 % |
+
+Rows with the final patch: Zen 3 at 8 threads, and N1 Qwen3 plus every N1 row at 32 threads. The
+other rows were measured with the first version. For MXFP4 and Q4_0 the two versions do the same
+work, and the activation quantizers have SIMD forms on both machines.
+
+**Token generation** (tg128) stays within the spread at every point, e.g. gpt-oss on N1 at 32
+threads 42.60 → 42.10, Qwen3 on Zen 3 at 8 threads 16.42 → 16.51. One N1 run at 32 threads read
+38.03 → 31.86 ± 3.76 for Qwen3. Repeated in alternating rounds on that host (load average 33 from
+other users), it read 38.00 / 35.10 and 32.20 / 35.11: noise.
+
+## The defect the real weights found
+
+The first version of the batched path quantized an expert's rows inside every (expert, column
+chunk) unit. With more threads there are more column chunks per expert (`nck`), so the same rows
+were quantized up to four times at 32 threads. On the N1, Qwen3-30B (Q4_K):
+
+| threads | upstream | first version | final |
+|--:|--:|--:|--:|
+| 8 | 56.66 | 62.22 (+10 %) | 66.77 (+18 %) |
+| 16 | 107.62 | 104.95 (−2 %) | 121.84 (+15 %) |
+| 32 | 200.91 | 164.04 (**−18 %**) | 215.69 (+13 %) |
+
+The upstream column of the final run is from a later series on the shared host: 56.81 / 105.99 /
+190.66.
+
+`perf` at 32 threads: upstream's path spends 36.5 % in `ggml_gemv_q4_K_8x4_q8_K` and 13.9 % in
+`ggml_gemv_q6_K_8x4_q8_K`. The first version spent 30.3 % in `ggml_gemm_q4_K_8x4_q8_K` and 5.5 % in
+the Q6_K gemm, but **20.8 % inside `forward_mul_mat_id` itself** (the inlined
+`ggml_quantize_mat_q8_K_4x4_generic`) plus 7.5 % in `memcpy`. x86 has an AVX2
+`ggml_quantize_mat_q8_K_4x8`; ARM has no SIMD Q8_K quantizer at all. MXFP4 and Q4_0 quantize to
+Q8_0, which has NEON forms, so only K-quants on ARM showed it.
+
+The final version gathers and quantizes every group of four rows once, into a buffer the units
+share (`work_size` reserves one row per routed token slot: 9.6 MB for Qwen3-30B at a 512-token
+batch). The groups are split statically over the threads, then a barrier; both are skipped when no
+expert got four tokens.
+
+## KLD: why Q4_K is not zero
+
+| model | machine | batched vs upstream: mean KLD | max | same top token |
+|---|---|--:|--:|--:|
+| gpt-oss-20b (MXFP4) | Zen 3 | 0.000000 | 0.000052 | 100 % |
+|  | N1 | 0.000000 | 0.000048 | 100 % |
+| OLMoE-1B-7B (Q4_0) | Zen 3 | 0.000000 | 0.000063 | 100 % |
+|  | N1 | 0.000000 | 0.000056 | 100 % |
+| Qwen3-30B-A3B (Q4_K) | Zen 3 | 0.007511 | 0.929 | 95.8 % |
+|  | N1 | 0.007316 | 0.897 | 96.1 % |
+
+For MXFP4 and Q4_0, gemm and gemv round alike, so batching changes nothing. For Q4_K they round
+differently (another summation order of the Q8_K block sums). In an MoE model, a router near a tie
+then picks another expert, which turns a rounding difference into a different token now and then.
+
+**The reference settles which path is "right".** Both paths were compared with the unrepacked
+generic computation (`--no-repack`, 8 chunks, N1):
+
+- upstream's repack path: mean KLD 0.007986, same top token 95.5 %;
+- the batched path: mean KLD 0.008496, same top token 95.0 %.
+
+Both are as far from it as from each other. The final version gives the same KLD as the first,
+digit for digit, since it computes the same products.
+
+## Through the jar
+
+`LlamaModel.setCpuMoeBatched(false/true)` alternated in one JVM on the same chat request, prompt
+cache off, the CI-built 6.1.0-SNAPSHOT `cpu-linux-aarch64` jar (first version of the patch), N1,
+16 threads:
+
+- gpt-oss-20b: prompt 79.4 / 81.1 → 130.7 / 130.6 t/s (+63 %), generation 29.9 → 29.3;
+- Qwen3-30B: 82.5 / 84.5 → 82.6 / 82.7 (the first version's Q4_K result at 16 threads).
+
+The switch reaches the native path through the jar, and the loader picked `armv8.2_2`.
+
+## s390x
+
+ggml repacks only Q4_0 on s390x (the VXE `q4_0_4x8` kernels), so OLMoE-1B-7B Q4_0, converted to
+big-endian with `gguf_convert_endian.py`, on cfarm191 (IBM z16, 8 CPUs, module `z16`; a shared host,
+load average 5 from other users during the run):
+
+| threads | pp512 upstream | batched | | tg128 upstream | batched |
+|--:|--:|--:|--:|--:|--:|
+| 2 | 24.34 | 38.97 | +60 % | 8.30 | 8.26 |
+| 4 | 45.79 | 76.23 | +66 % | 17.04 | 17.37 |
+| 8 | 43.33 | 59.03 | +36 % | 0.31 | 0.33 |
+
+At 8 threads every CPU was shared with other users' processes, which spinning threads punish:
+generation collapsed on both paths alike. KLD of the batched path against upstream's (8 threads,
+20 chunks): mean 0.000000, max 0.000059, same top token 99.98 %.
+
+## Not measured here
+
+AVX-512/AMX (`sapphirerapids`) and Zen 4 (`zen4`), which need a paid cloud instance; ARM with i8mm
+(Graviton 3/4: the `q4_K_8x8` / `q4_0_4x8` kernels); and the hybrid-core case the original
+measurement emulated.