Repository navigation
HIP decode: the 4-bit stream tile for MTP verify rows (scale/bias layout, lanes at K=640, row reuse) #1604
Description
On AMD Ryzen AI MAX+ 395, Radeon 8060S gfx1151 (40 CUs, 128 GB unified memory, measured 225.6 GB/s read), Ubuntu 26.04, archive HIP 7.1 (clang 21), TensorFold 7c068895, TensorFold/Qwen3.8-Flash-Next-MLX-4bit-MTP.
The 4-bit affine_dot2_stream decode tiles carry 77% of an MTP verify round. Measured: double-buffering the multi-row loop is 9% slower (the existing choice to give registers to the rows' activations is right). Open candidates, each to be A/B'd on served decode: scale and bias read column-major (scale[col * groups + g]) as two separate arrays; lanes idle at K=640 (routed down) under the 32-lane rule; a 2-row x 8-column variant for 4 verify rows; rows of a routed item that hold one token computing all four.
Proposal
Land only the candidates that improve served decode with byte-identical tokens; report the rejected ones with numbers.
Activity
gilby commentedon Oct 29, 2026
Measured on AMD Ryzen AI MAX+ 395 (Radeon 8060S
gfx1151), TensorFold7c068895,TensorFold/Qwen3.8-Flash-Next-MLX-4bit-MTP, MTP 3 drafts,--parallel 1 --context 131072,tools/bench_openai.pymedians (code / chat tok/s). Stock: 49.98 / 43.95.Not pursued, from reading the code: scale and bias are already read contiguously per column (
sb[c] + (q >> gshift)), so an interleaved layout needs a load-time repack for little expected gain;take_itemalready setsa.mto an item's token count and the stream loop stops at it, so dead rows only remain inside one 4-row group (ALU on a memory-bound kernel). A trace of a 442-token reply puts allaffine_dot2_streamvariants at 47.2 ms of a 61.5 ms verify round, 170-182 GB/s against 225.6 GB/s measured, so these tiles are near the bandwidth limit; the rest of the round is about 850 small launches (#1603, router part in #1614). Nothing to land here for now.ashhart commentedon Oct 30, 2026
Thank you, @gilby, for measuring the three affine_dot2_stream candidates on gfx1151 and writing up why each was rejected. With the tiles near the bandwidth limit, we agree there is nothing to land here. The remaining cost is the roughly 850 small launches, which #1603 and the router work in #1614 pick up, so we're closing this one. Please reopen if a candidate turns up that beats the current tile.