Repository navigation
memory/encoder graph nar qwen3 - #224
Merged
Merged
Conversation
The sched-scratch line from #217 says how big a graph's committed buffer is; it does not say which tensors make it that big. Finding that out meant bisecting a graph by hand. TRANSCRIBE_GRAPH_TOP=N logs the N largest nodes of every graph allocated through alloc_sched_graph: op, type, shape, assigned backend and name, sorted by size, plus the total if nothing were reused (an upper bound; the real buffer is the live set). Views are skipped since they own no memory. Unset or non-positive is inert, so this is a debugging aid with no behaviour change. It is what attributed both memory branches that follow: granite_nar's [100352, T_enc] BPE CTC head and qwen3_asr's per-chunk conv activations were invisible in the scratch totals alone.
The encoder was the whole memory story: 6.0 GB of committed Metal scratch for an 811 MB model at 5 minutes, 8.0 GB RSS on CPU, growing ~linearly with audio length. Two causes, both measured with TRANSCRIBE_GRAPH_TOP: 1. The 3x Conv2d subsampler ran every chunk in one batch. Per chunk that is 6.1 MB of conv0 activation and 6.9 MB of conv1 im2col, so ~1.2 GB of scratch per minute of audio. Run it in groups of 16 chunks inside the one graph instead: each group's intermediates retire before the next group's are allocated, and only the [d_model, T_per_chunk, g] outputs are concatenated. Per-chunk results do not depend on the group size, and enc.subsample.out / enc.pos_add.out are bit-identical to before. Shared by the single-shot and batched graphs (subsample_chunks). 2. Encoder flash was off (encoder_use_flash = false since 1d244e5, a Vulkan perf commit with no recorded reason), so every block materialised [T_enc, T_enc, n_heads] f32 scores: T^2, 0.9 GB at 5 minutes and 13.6 GB at 20. Cause 1 hid it at 5 minutes. Flash is GQA/bidirectional-native here and needs no mask, which also drops the all-zero [T_enc, T_enc] mask that was built host-side and uploaded per run. Metal, Q8_0, dots-full: scratch 6051 -> 397 MB at 306 s, and 1224 s now completes at 1797 MB (previously not runnable). CPU RSS at 306 s 8035 -> 2281 MB. Encode 2843 -> 2120 ms on Metal; CPU encode unchanged (1283 -> 1284 ms at 60 s), so the flash default is safe on both. Transcripts are byte-identical to main at 60/120/306 s, and LibriSpeech test-clean Q8_0, b8, 2620 utts is 2.13% WER against the catalog's published 2.11% [1.95, 2.27]. TRANSCRIBE_NO_FLASH still selects the old attention path.
The encoder committed 6.5 GB of scratch at 306 s for a 2.4 GB model and the
run then rejected the input anyway; at 240 s it was 11.1 GB RSS. Three
causes, measured with TRANSCRIBE_GRAPH_TOP:
1. The BPE CTC head ran on every frame: [100352, T_enc] f32, i.e. 6.1 GB at
306 s (50 encoder frames per second), then copied to the host whole. The
reference pools it over windows of 4 frames with importance weights that
sum to 1, and a head is affine:
sum_t w_t (W h_t + b) = W (sum_t w_t h_t) + b
so pool the 1024-wide hidden states first, then run the head on
n_windows = T_enc / 4 columns. Tiles of 256 windows bound the logits to
~103 MB, and ggml_argmax runs in-graph, so only one id per window (plus
each window's importance mass, for the all-blank rule) reaches the host.
Same arithmetic, 4x fewer FLOPs in the head. collapse_bpe_window_ids
keeps compute_bpe_ctc_initial_hypothesis's exact semantics, including
both blank-id schemes.
2. The depthwise conv ran through conv_1d_dw_f32's im2col + mul_mat, which
inflates the activation by conv_kernel (15) to feed a matmul whose K is
that same kernel: [15, T, 2048] f32, 1.9 GB at 306 s. Use
conv_2d_dw_direct_f32, as granite5_ctc already does.
3. The decoder built [T_total, T_total, 16] f32 scores (539 MB at 180 s,
1.07 GB at the 4096-token cap) plus the K/V ggml_repeat copies, and
wrote the whole [vocab, n_text] logits tensor back to the host. Flash
attention is GQA-native and needs no mask here (bidirectional); the LM
head runs in tiles of 256 text rows with the argmax in-graph. The full
logits form is still built when tensor dumping is on, since
dec.text_logits is a parity tap. Decode 5.20 -> 4.16 s at 180 s.
Metal, Q8_0: 180 s 8369 -> 1032 MB RSS, 8849 -> 1039 MB footprint, scratch
3832 -> 708 MB; 240 s 11146 -> 1339 MB RSS. 306 s still rejects
(3059 audio + 1893 text tokens > 4096), which is the model's context, not
memory. Transcripts identical to main at 60/120/180/240 s and 0/647
differ on fleurs-en; LibriSpeech test-clean Q8_0, b8, 2620 utts is 1.28%
WER against the catalog's published 1.29%. Branch-vs-main tensor drift
enters at enc.block.0.post_conv (2.4e-4, the direct-depthwise accumulation
order) and reaches enc.block.15.out 0.25 max / 0.019 mean; the reference
tolerances are Q8_0-limited on main too (2/15 either way).
Review follow-up on the two memory commits. - qwen3_asr::build_cu_seqlens_mask has no caller: the single-shot encoder passes mask=nullptr. An unused implementation can drift from the real path, so delete it and move its reasoning (the measured eager reference attends unmasked; vLLM's flash-attn-2 windowing scores worse) into the header's shape note, which also stops claiming mask_in is [T_enc, T_enc] when it is now always null. - granite_nar::compute_bpe_ctc_initial_hypothesis still has no production caller either, but it is the reference formulation the pooled in-graph head claims to reproduce, so it earns a test instead of a comment: tests/granite_nar_bpe_pool_unit.cpp builds the same affine head both ways (frame-level logits + host pool vs importance-weighted hidden pool + head + argmax) and requires identical tokens, including the all-blank-window rule, the partial tail window and both blank-id schemes. It also pins the collapse rules directly. - collapse_bpe_window_ids no longer truncates on a length mismatch: the ids and masses both come from tensors sized eb.n_bpe_windows, so a mismatch is a graph/caller bug. It now returns false and run() fails with TRANSCRIBE_ERR_BACKEND and a message naming both lengths, instead of emitting a silently shorter hypothesis. - tests/tolerances/granite_nar.json pointed at the old function. No behaviour change on the model path: transcripts byte-identical to the previous commit at 60/180 s (granite_nar) and 120 s (qwen3_asr), ctest 89/89.
…annot overflow the graph
…lumbing Review follow-ups on top of b490c12: - Remove TRANSCRIBE_QWEN3_ASR_CONV_PASS; the pass size is the constant kConvPassChunks. - Remove TRANSCRIBE_GRAPH_TOP (undocumented, unused outside manual debugging); transcribe-backend.{h,cpp} are back to main. - qwen3_asr: drop the always-null single-shot EncoderBuild::mask_in, and the orphaned build_cu_seqlens_mask doc comment and mask-upload note. - Tolerance notes: granite_nar decoder attention is flash by default and the encoder depthwise conv is the direct op; qwen3_asr encoder attention is flash by default. Re-measured on CPU (validate.py all, BF16): granite_nar 15/15, qwen3_asr 0.6B 13/13 within tolerance, transcripts match the reference. - clang-format the qwen3_asr conv-pass change and tests/granite_nar_bpe_pool_unit.cpp (the format gate failed on both).
cunba-ai
pushed a commit
to cunba-ai/transcribe.cpp
that referenced
this pull request
Oct 27, 2026
…r#220, handy-computer#223, handy-computer#222, handy-computer#225, handy-computer#224) - handy-computer#220 causal_lm: alias the packed gate+up projection onto the existing weights (no duplicate conversion copies at load) - handy-computer#223 scripts/bench: memory.py peak-memory harness - handy-computer#222 moonshine_streaming: bounded encoder memory past the 82 s training window and an out-of-bounds read fix; input-cap smoke replaces the truncation test - handy-computer#225 catalog: Apple M4 speed cells published; weak-machine targets capped - handy-computer#224 qwen3-asr nar: encoder graph memory bounded Fork note: no fork-owned files in the conflict set.
cunba-ai
pushed a commit
to cunba-ai/transcribe.cpp
that referenced
this pull request
Oct 27, 2026
transcribe-e2e (tools/, workspace member) loads the fork's self-contained transcribe.dll exactly the way sound-server does — the transcribe-cpp safe wrapper with the dynload feature, runtime dlopen, no compile-time native dependency — and runs 23 checks over the public safe API only: - library: init_dynamic full-symbol resolve, >=0.5.0 version gate, backend/device enumeration, embedded-VAD probe - ASR core (whisper-tiny): offline run, no_mmap parity (0.4.1 mmap via dynload), run_batch == serial, progress monotonicity + cancel-at-first- tick -> Aborted with partial transcript, de->en translate - families + 0.4.1 behavior: moonshine, sensevoice ITN default ON, fun-asr-mlt ITN default ON (handy-computer#205), qwen3 NAR (handy-computer#224), gigaam windowed 33.8 s Russian (handy-computer#208), parakeet v3 [256,256] local attention (handy-computer#199), multitalker bundle diarize=ON (handy-computer#219, bundle composed on demand via compose-multitalker-bundle.py) - streaming: buffered stream == offline (multitalker streaming ckpt), moonshine-streaming feed/finalize (handy-computer#222) - sortformer: DIARIZE role + LowLatency preset, push-stream session with tentative->committed convergence (fork SFPS) - embedded VAD: one-shot segments, VAD-prechunk -> whisper run_batch (the sound-rs pipeline), stream_feed/flush prob parity Check audio reuses the committed oracle mixes (sortformer-2spk-mix.wav, multitalker-2spk-mix.wav) so results stay comparable with the pinned goldens in tests/. Missing model files SKIP; a FAIL exits 1. scripts/e2e/run-e2e.ps1 drives the whole flow: fetch+verify the embedded VAD asset (sha256 single-sourced from cmake), build the SHARED_EMBED DLL, compose the multitalker bundle, build the CLI, run the matrix. Validated live: 23 pass / 0 skip / 0 fail on the merged 0.4.1 tree. Generated-by: zcode
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.