Repository navigation
moonshine_streaming: bound encoder memory, and fix an out-of-bounds read past 82 s - #222
Merged
Merged
Conversation
The one-shot (and batch) encoder ran the whole utterance as one graph: one
dense [T_enc, T_enc] f32 mask per layer on host and device, plus
[T_enc, T_enc, n_heads] attention scores. Memory grew with T^2 - 12 / 20 /
26 GB of footprint for tiny / small / medium on 5 minutes of audio.
Encode in windows of 1024 emitted frames (~20 s) instead, each carrying the
encoder's full receptive field: cumulative_left_context + the frontend conv
pad on the left, cumulative_right_context on the right. Every layer attends
through a finite sliding window and the convs are causal, so this is exact in
the mathematical sense - the same algebra the streaming path already relies on
and parity-tests. Window cuts sit on encoder-frame boundaries so both strided
convs stay phase-aligned. Inputs that fit one window take the old single-pass
path unchanged.
Not bit-exact: Metal's kernels order reductions by a frame's position in the
graph, so windows that don't start at frame 0 differ at ~5e-4 of the
activation range, uniformly, with no rise at the cuts. Shortening the left
context by 10 or 30 frames produces 1e-2 / 4e-1 spikes at every cut, so the
check would catch a missing-context bug. The residual is about half of what
toggling flash attention does to the single pass.
Peak footprint, Metal, Q8_0, inside the model's 81.9 s input range:
30 s 60 s 80 s
tiny 303 -> 240 669 -> 283 1040 -> 305
small 510 -> 413 1125 -> 515 1734 -> 561
medium 742 -> 609 1612 -> 786 2465 -> 863
Transcripts are byte-identical to main on jfk and 30/60/80/120 s for all three
sizes; stream_parity passes with and without a forced 64-frame window.
TRANSCRIBE_MOONSHINE_STREAMING_ENC_WINDOW overrides the window size.
Beyond 81.9 s (4096 encoder frames) the adapter's pos_emb table runs out:
that is a separate, pre-existing bug (OOB get_rows: CPU aborts, Metal reads
garbage) and not fixed here.
The adapter adds pos_emb[frame] to every encoder frame from a table of
dec_max_position_embeddings (4096) rows - 4096 encoder frames, 81.92 s of
audio per decode. The reference raises IndexError past it. The port had no
bound, so a longer input fed frame indices >= 4096 to ggml_get_rows:
- CPU: GGML_ASSERT(i01 >= 0 && i01 < ne01) aborts the process (rc 134), for
one-shot input over 82 s and for any stream that runs past 82 s - a host
app dictating for a minute and a half on a CPU backend crashes.
- Metal: silent out-of-bounds reads, i.e. garbage transcripts (<unk>,
repetition loops on long files).
Enforce the cap with the contract docs/input-limits.md already defines for an
absolute position wall (voxtral_realtime):
- transcribe_run: input whose padded length needs more than 4096 encoder
frames is rejected with TRANSCRIBE_ERR_INPUT_TOO_LONG before encoding.
- transcribe_run_batch: the same, per utterance (the batched GPU path now
carries a per-row fail status instead of hardcoding INVALID_ARG); the
serial path inherits it from run().
- Streaming: emission stops at frame 4096. The stream is flagged
transcribe_was_truncated() with a single WARN, later audio is dropped (and
not buffered), and feeds/finalize keep returning TRANSCRIBE_OK. A stream
that ends inside the last right-context window past the cap is handled at
finalize the same way.
- apply_adapter_window refuses any frame range outside the table, as the
backstop that keeps get_rows in bounds whatever a caller does.
- capabilities.max_audio_ms reports 81920 (it advertised ~1024 s, derived
from the decoder token cap, which is not the binding limit).
Docs move moonshine_streaming from the soft-window bucket ("output-bound,
~17 min") to the hard-cap bucket and spell out the streaming behaviour.
Verified on tiny/small/medium: 4096 frames (81.92 s) runs, 4097 is rejected,
on Metal and CPU; a 120 s stream on CPU (which aborted before) finalizes OK,
truncated, and its transcript equals one-shot over the first 81.92 s.
tests/moonshine_streaming_batch_truncation is replaced by
tests/moonshine_streaming_input_cap: its long row (love-loss.wav, 197 s) only
"truncated" because the decode was running on out-of-bounds positions. The new
gated test pins the boundary, one-shot rejection, the per-row batch status, and
a stream past the cap (exact transcript on the default backend, word for word
on CPU, whose shape-dependent kernels can move a comma), on both backends.
- Compare the streamed transcript to one-shot exactly on the default backend (byte for byte proves the stream consumed the first 81.92 s and nothing else); on CPU assert only the invariants, since ggml's shape-dependent kernels can move a comma between a windowed and a single-pass run. Drops the word-level comparison helper, which was the weak part of the test. - Feed 90 s instead of the whole 306 s sample: enough to cross the cap. (The runtime is dominated by the at-cap encode's masks either way, ~37 s.)
cunba-ai
pushed a commit
to cunba-ai/transcribe.cpp
that referenced
this pull request
Oct 27, 2026
…r#220, handy-computer#223, handy-computer#222, handy-computer#225, handy-computer#224) - handy-computer#220 causal_lm: alias the packed gate+up projection onto the existing weights (no duplicate conversion copies at load) - handy-computer#223 scripts/bench: memory.py peak-memory harness - handy-computer#222 moonshine_streaming: bounded encoder memory past the 82 s training window and an out-of-bounds read fix; input-cap smoke replaces the truncation test - handy-computer#225 catalog: Apple M4 speed cells published; weak-machine targets capped - handy-computer#224 qwen3-asr nar: encoder graph memory bounded Fork note: no fork-owned files in the conflict set.
cunba-ai
pushed a commit
to cunba-ai/transcribe.cpp
that referenced
this pull request
Oct 27, 2026
transcribe-e2e (tools/, workspace member) loads the fork's self-contained transcribe.dll exactly the way sound-server does — the transcribe-cpp safe wrapper with the dynload feature, runtime dlopen, no compile-time native dependency — and runs 23 checks over the public safe API only: - library: init_dynamic full-symbol resolve, >=0.5.0 version gate, backend/device enumeration, embedded-VAD probe - ASR core (whisper-tiny): offline run, no_mmap parity (0.4.1 mmap via dynload), run_batch == serial, progress monotonicity + cancel-at-first- tick -> Aborted with partial transcript, de->en translate - families + 0.4.1 behavior: moonshine, sensevoice ITN default ON, fun-asr-mlt ITN default ON (handy-computer#205), qwen3 NAR (handy-computer#224), gigaam windowed 33.8 s Russian (handy-computer#208), parakeet v3 [256,256] local attention (handy-computer#199), multitalker bundle diarize=ON (handy-computer#219, bundle composed on demand via compose-multitalker-bundle.py) - streaming: buffered stream == offline (multitalker streaming ckpt), moonshine-streaming feed/finalize (handy-computer#222) - sortformer: DIARIZE role + LowLatency preset, push-stream session with tentative->committed convergence (fork SFPS) - embedded VAD: one-shot segments, VAD-prechunk -> whisper run_batch (the sound-rs pipeline), stream_feed/flush prob parity Check audio reuses the committed oracle mixes (sortformer-2spk-mix.wav, multitalker-2spk-mix.wav) so results stay comparable with the pinned goldens in tests/. Missing model files SKIP; a FAIL exits 1. scripts/e2e/run-e2e.ps1 drives the whole flow: fetch+verify the embedded VAD asset (sha256 single-sourced from cmake), build the SHARED_EMBED DLL, compose the multitalker bundle, build the CLI, run the matrix. Validated live: 23 pass / 0 skip / 0 fail on the merged 0.4.1 tree. Generated-by: zcode
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
actually cap moonshine and also reduce its mem usage