[joseedson18jc]

Problem and result

In a speculative window of 3 or 4 tokens, moe_gu's 3- and 4-entry expert groups interleave gate and up rows. This adds gu_body_seq behind function constant 25 (MOE_SEQ). It decodes an expert's gate units and accumulates them, then its up units: one pass over the weights, same math order. The host sets it only when all of these hold:

• the window has 3+ tokens;
• the tile has gr == 1;
• gate/up is Q2_0, IQ2_XXS or IQ3_XXS.

1- and 2-token pipelines are built without the path. With it in the same pipeline, the M1 Max ran 1 and 2 tokens 8-10% slower.

Gate/up kernel time, µs a layer, expert_parity --bench --dn Q2_0 --share 0.9, M1 Max, single runs, without seq → with seq:

gate/up	tile	3 tokens	4 tokens
Q2_0	32,1,4 (main's tile)	198.8 → 186.9 (−6%)	265.7 → 261.0 (−2%)
IQ3_XXS	32,1,4	247.2 → 239.9 (−3%)	319.4 → 307.3 (−4%)
IQ2_XXS	32,1,4	229.9 → 228.5 (−1%)	295.4 → 286.3 (−3%)
Q2_0	8,1,2	164.8 → 111.2 (−33%)	216.8 → 144.1 (−34%)
Q2_0	16,1,4	187.8 → 143.0 (−24%)	249.2 → 188.3 (−24%)
IQ3_XXS	8,1,2	197.2 → 171.9 (−13%)	248.4 → 207.8 (−16%)
IQ2_XXS	8,1,2	184.9 → 163.1 (−12%)	232.4 → 203.0 (−13%)

On main's 32,1,4 tile the gains are small. The 1-2% ones are within noise. The large gains need the 8,1,2 or 16,1,4 tiles, which are not on main. They come from the opt-in STRATA_METAL_MOE_GU in #4272.

Why the gate is narrow: the same path on M1 Max was 10-43% slower for Q4_K, Q5_K, Q6_K, Q8_0, IQ4_XS and IQ2_XS at 32,1,4, and for Q2_0 at 32,2,4 (189 → 270 µs at 3 tokens).

Related discussion

Builds on the tiles of #4272. Each PR stands alone: with only this one, seq runs at 32,1,4. gr == 1 keeps it off any 2-row tile.

Approach

• metal/kernels/moe.metal:
  • adds gu_body_seq<F,LPR,NR,NE> and MOE_SEQ (function constant 25);
  • moe_gu cases 3 and 4, and the chunk loop's m == 3 / m == 4, take it when set.
• metal/src/engine/window.cpp: the decode site sets constant 25 under the gate above, with a #seq pipeline cache key.
• metal/tests/expert_parity.cpp: reads MOE_SEQ and PARITY_SHARE (default 0.3) from the environment.

Validation

Environment: M1 Max, macOS 27.0, revision 57e4d4d on main 5e8a4cc.

cmake flags: -G Ninja -DCMAKE_BUILD_TYPE=Release, Apple clang, metal-cpp from third_party.

Passed

• Build, rc 0.
• Parity with seq: MOE_SEQ=1 expert_parity --gu {Q2_0,IQ2_XXS,IQ3_XXS} --dn Q2_0 --gucfg 32,1,4 --ntok {3,4,6,8}. All 12 PASS; h and y rel L2 ~1e-7. ntok 6 and 8 cover the chunk loop.
• Parity with seq at 8,1,2 and 32,2,4 for Q2_0, ntok 3/4/6/8: PASS.
• Parity with seq for Q8_0, and for Q4_K, IQ3_S, Q5_K and IQ4_XS gate/up with Q2_0 down: PASS. Errors are identical with and without seq.

Failed

• expert_parity crashes (rc 138) when gate/up and down are both a K- or i-quant. This also happens on the unpatched build, which is why the runs above use --dn Q2_0.

Not run

• No end-to-end run on a main build.
• On a downstream test branch with 8,1,2 tiles, a 35B-A3B Q2_0 model at --spec 2 showed no tok/s gain (43.0 vs 42.1 sampled, 43.2 vs 43.9 greedy). --spec 3 was slower there for other reasons.
• --spec 4 and other Apple chips were not measured.

Limits

• Kernel-level change, measured on M1 Max only; single runs, no repeats.
• With 8,1,2, 2 tokens ran +1.2% slower. That held with the run order reversed, though the 1- and 2-token pipelines do not take the seq path.
• 1 and 2 tokens were not measured at 32,1,4.
• Trellis gate/up types are not gated in.