Repository navigation
causal_lm: alias packed gate+up onto the existing weights - #220
Merged
Merged
Conversation
pack_gate_up allocated a fused [hidden, 2*intermediate] tensor per decoder
block and copied gate's bytes followed by up's into it, while the originals
stayed live. Every packed family therefore held its FFN gate+up weights twice:
1917 MB (voxtral-realtime), 1530 (voxtral-mini-3b), 714 (canary-qwen), 3x688
(granite), 4x178 (moss, funasr-nano, qwen3-asr) - ~7.5 GB in total.
The converter already writes ffn.up.weight straight after ffn.gate.weight, and
concat-along-dim-1 is byte-concat for row-wise quants, so the fused tensor is
already in the file. Both load paths preserve that: a copied buffer is filled
in catalog order, and load_common's mmap points each tensor at its file offset.
So when gate and up are adjacent in one backend buffer, allocate the fused
tensor at gate's address instead and copy nothing. Otherwise - a device that
pads tensor storage, a mapping split into views, tensors in different buffers -
fall back to the copying path unchanged. TRANSCRIBE_NO_GATE_UP_ALIAS forces the
copy path for A/B measurement.
Bytes, dtype, shape and kernel are unchanged, so results are unchanged: every
transcript below is byte-identical between the two paths, on Metal with mmap,
on CPU, and with --no-mmap.
Peak RSS, Q8_0, Metal, warm runs, 306 s dots-full (packed = duplicate removed):
variant packed before after delta
granite-4.0-1b-speech 680 2273 905 1368
granite-speech-4.1-2b 680 2273 904 1369
granite-speech-4.1-2b-plus 680 2392 1024 1368
canary-qwen-2.5b 714 3331 1890 1441
qwen3-asr-0.6b 178 1675 1317 358
voxtral-mini-3b 1530 4870 1804 3066
voxtral-realtime 1492 5057 2148 2909
moss 178 2156 1800 356
funasr-nano 178 1534 1176 358
The delta is ~2x the packed bytes under mmap because the mapped file pages are
resident either way; the anonymous reduction is the packed bytes, confirmed with
--no-mmap (granite: 4041 -> 3362). Wall time is unchanged everywhere (19.46 ->
19.41 s, 13.86 -> 13.76 s, 125.35 -> 125.26 s); only a cold first run pays the
page faults. funasr-nano on dots-full trips the repetition guard and exits
non-zero in both modes with identical text.
voxtral-small-24b (25.81 GB Q8_0 file, M4 Max, same sample) aliases 40/40 as
well, and the effect there is much larger than the packed bytes because the
whole model stays file-backed:
peak RSS 2.30 GiB aliased / 29.03 GiB copied
peak footprint 2.65 GB aliased / 17.1 GB copied (macOS reclaims mapped pages
the run does not touch)
Transcripts are byte-identical across all four runs. The one case the alias path
cannot serve is a file larger than the device buffer type's max size - only then
is a mapping split into several views - where it declines and copies, i.e. the
pre-branch behaviour. That did not reproduce on the M4 Max (max size 80.64 GiB
for a 25.81 GB file): the guard is for smaller devices, not a known regression.
Not covered here: the other quant variants, qwen3-asr 1.7b and fun-asr-mlt-nano
(no GGUFs checked out).
cjpais
force-pushed
the
memory/pack-gate-up-alias
branch
from
October 20, 2026 06:30
a104876 to
bd2355e
Compare
Two review hardenings on the aliasing path: - gate_up_is_adjacent now requires nbytes(fused) == nbytes(gate) + nbytes(up) before aliasing. The copying path caught a gate/up shape mismatch for free, because it allocated the fused tensor and then compared sizes while filling it; an aliased block skipped that, so a mismatched pair (adjacent but not the expected shape) would have run on a short tensor instead of failing. - qwen3_asr_smoke clears TRANSCRIBE_NO_GATE_UP_ALIAS before loading. The test asserts the default (aliased) configuration, so a caller exporting the fork switch - as CI or a bisect might - would have failed it spuriously.
Metal decides to split a mapping after rounding its length up to the page size, so a buffer whose unrounded size only just fits under the buffer type's max size could still be split - and an alias across a split boundary hands the kernel the wrong Metal buffer. Decline with a page of slack instead of an exact comparison.
Review follow-ups on the aliasing branch: - The size check (nbytes(fused) == nbytes(gate) + nbytes(up)) ran only while filling a copied block. gate_up_is_adjacent's own adjacency test makes a mismatch impossible to alias, but the promise in the header is an error, and the failure should be reported the same way whether or not a block happens to be adjacent. Move it into the validation loop, before the alias/copy split, and drop the copy-path copy of the check. - README claimed the fusion always costs no extra weight memory. It falls back to copying when the device layout does not allow aliasing (padded tensor storage, or a mapping split because the file exceeds the device max buffer size). Say so, and say 'usually'. - qwen3_asr_smoke used setenv/unsetenv directly, which MSVC does not have. Use the same _putenv_s/setenv helper shape as weights_mmap_smoke.
- tests/pack_gate_up_unit.cpp: covers alias (adjacent in one buffer), copy (a gap between gate and up), mixed (one adjacent block and one gapped block in the same call, which nothing did before), the env fork switch, and the shape-mismatch error. Built by hand on the CPU backend, so it pins the behaviour without a model and without a fixture whose tensor order is incidental. It also pins the ownership rule: the aliased path allocates no buffer and free() must leave the weights buffer usable. - The fallback summary is now INFO instead of DEBUG and names the reason, e.g. "gate+up packed by copying (40 blocks; disabled by TRANSCRIBE_NO_GATE_UP_ALIAS x40)". A converter that reordered gate/up would silently give the memory win back otherwise: no wrong answers, nothing in any test that used a hand-generated fixture, and no fixture can stand in for the real converter here. - qwen3_asr_real_smoke_0_6b asserts the fusion on real converter output: when the weights are mapped, the packed tensor must alias gate, and when they are copied the fused bytes must still be gate followed by up. - The comment in pack_gate_up pointed at a check_shapes() that does not exist; name the GET_LIN shape checks in each family's weights.cpp instead. - qwen3_asr_real_smoke_0_6b includes ggml-backend.h for the tensor reads it now does.
cjpais
force-pushed
the
memory/pack-gate-up-alias
branch
from
October 20, 2026 07:13
731a00e to
f4108f6
Compare
cunba-ai
pushed a commit
to cunba-ai/transcribe.cpp
that referenced
this pull request
Oct 27, 2026
…r#220, handy-computer#223, handy-computer#222, handy-computer#225, handy-computer#224) - handy-computer#220 causal_lm: alias the packed gate+up projection onto the existing weights (no duplicate conversion copies at load) - handy-computer#223 scripts/bench: memory.py peak-memory harness - handy-computer#222 moonshine_streaming: bounded encoder memory past the 82 s training window and an out-of-bounds read fix; input-cap smoke replaces the truncation test - handy-computer#225 catalog: Apple M4 speed cells published; weak-machine targets capped - handy-computer#224 qwen3-asr nar: encoder graph memory bounded Fork note: no fork-owned files in the conflict set.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
should help reduce rss