Skip to content

refactor: replace the vendored zip and Vorbis code with own implementations - #2950

Merged
Danielku15 merged 8 commits into
developfrom
refactor/vorbis-own-decoder
Oct 11, 2026
Merged

Danielku15 merged 8 commits into
developfrom
refactor/vorbis-own-decoder

Conversation

@Danielku15

Copy link
Copy Markdown
Member

Note

AI-authored disclosure (alphatab-ai-authored-v1)

Portions of this content were authored by an AI agent. The agent has read
AGENTS.md and the human submitter accepts responsibility for
compliance with the rules in that document.

Issues

Fixes #2948

Proposed changes

alphaTab shipped ports of three libraries for its low-level algorithms:

  • Inflate: the Haxe Standard Library inflater.
  • Deflate: SharpZipLib.
  • SF3 sample decoding: NVorbis, which itself contains libvorbis code.

This PR replaces them with alphaTab-owned implementations written from RFC 1951 and the Vorbis I specification, plus standard published algorithms (Moffat–Katajainen code lengths, a DCT-IV based IMDCT). The four library notices are removed from LICENSE.header. zlib and libvorbis are only used as references in tests. TinySoundFont stays, as planned in the issue.

The commits are split per step: zip (CRC-32 fix, inflater, deflater, licence), then Vorbis (IMDCT, decoder, robustness tests, licence).

Zip (src/zip)

  • DeflateFormat: the parts shared by both directions. These are the length/distance tables, the code length order, the fixed code and the canonical code assignment.
  • Inflater (replaces Inflate, HuffTools, Huffman):
    • decodes a whole stream into one growing buffer with single-level lookup tables;
    • reads input bytes only when needed, so it stops exactly at the end of the stream;
    • validates the data as strictly as zlib.
  • Deflater (replaces Deflater, DeflaterEngine, DeflaterHuffman, DeflaterConstants, PendingBuffer and the unused Adler32):
    • compresses a whole buffer with hash chains and lazy matching, using zlib's default search limits;
    • per block, it picks the smallest of stored, fixed and dynamic encoding;
    • dynamic codes are length-limited Huffman codes computed with Moffat–Katajainen;
    • the output is byte-identical on all platforms.
  • ZipWriter computes the CRC-32 itself.

Vorbis (src/synth/vorbis)

  • OggReader: concatenates the page bodies of the first logical stream. Packet boundaries come from the lacing values alone, so packets spanning pages need no special handling.
  • VorbisBitReader: reads least significant bit first. Reads past the end of a packet return zero bits and set an end-of-packet flag. A codeword that fits into the remaining bits still decodes.
  • VorbisCodebook:
    • Codewords are assigned in a binary tree. Each node knows the shallowest free depth below it, so each insertion walks a single path, and lengths up to 32 bits need no 32-bit integers.
    • Decoding uses a 10-bit lookup table with a tree walk for longer codes.
  • VorbisFloor (floor 1), VorbisResidue (types 0/1/2), VorbisSetup (setup header, mappings, modes).
  • Mdct: the inverse MDCT as a DCT-IV over an n/4-point complex FFT. On its own it is 1.5–2× faster than the previous unrolled implementation.
  • VorbisDecoder:
    • the packet pipeline, windowing, overlap-add and granule trimming;
    • an instance caches the parsed setups (keyed by channels, block sizes and setup bytes), the MDCT tables and the window slopes. Hydra keeps one instance, so SF3 samples sharing a setup parse it once (default.sf3: 182 samples, 3 setups);
    • Hydra passes the byte range of sampleData instead of slicing a copy.

Defects of the previous code that are fixed

Area Previous behaviour Effect
CRC-32 The table and state used unsigned 32-bit values and >>>. C# and Kotlin convert these wrongly. Zip files exported on .NET/Android (e.g. GP7 export) had wrong checksums. On C# the check value of 123456789 came out as -2139127681 instead of -873187034.
Vorbis Codewords ending inside the Huffman lookahead at the end of a packet were dropped. libvorbis' bitrate management produces such truncated packets. The last residue values were missing in 909 of 920 GeneralUserGS samples. Worst deviation from libvorbis: −36 dBFS ("Distortion Gt-E2").
Vorbis Some streams were not cut at the final granule position. Encoder padding was played at the end, e.g. "Cello-G#1" had 37376 instead of 37067 frames.
Ogg Packets spanning pages lost their continuation, and further packets on such pages were dropped. Gaps in such streams. None of the SF3 files we checked contains them.
Vorbis The IMDCT was wrong for block sizes 64 and 128. Garbage output for such streams (valid in Vorbis I, unused by libvorbis' standard modes).
Vorbis Indices from the file were not validated: blocksize 1 was never checked, floor 1 values weren't clamped, book indices weren't range-checked. On corrupt data: NaN audio in JS, exceptions other than FormatError in C#/Kotlin.
Vorbis Residue 0/1 decoded channels marked "do not decode", and every submap got all channels. Wrong output for multichannel residue 0/1 or multi-submap streams.

Behaviour changes

  • Zip output: compressed data is about 10% smaller and its bytes differ from before, so exported GP7 files change.
  • Vorbis output: decoded samples now match libvorbis to float rounding (differences around 1e-7), so the audio changes slightly.
  • Errors: invalid Vorbis streams throw FormatError. Before, they threw a TypeError or an AlphaTabError.
  • Floor type 0 is not supported (FormatError). Only encoders before 2002 used it, no SF3 tool writes it, and stb_vorbis leaves it out as well.

C# and Kotlin

Both rewrites follow integer rules documented in DeflateFormat and VorbisBitReader:

  • values in bit operations stay in the int32 range;
  • no >>> and no compound bit assignments;
  • bit buffers stay below 32 bits;
  • typed arrays only receive in-range values;
  • no writes through subarray;
  • every index taken from the file is validated or clamped.

Building locally: run npm run clean --workspace=packages/csharp and npm run clean --workspace=packages/kotlin first. Stale generated files of the deleted classes otherwise collide.

Tests and verification

  • Zip (test/zip/Deflate.test.ts):
    • CRC-32 known answers;
    • inflater cases: zlib streams, sync flush, the edge cases of distance codes, stopping exactly at the end of the stream, size limits, invalid data;
    • deflater round trips (empty, runs, far repeats, incompressible, skewed frequencies), code length limiting, and a golden test with exact expected output on all three platforms.
  • Zip interop (test/zip/DeflateInterop.test.ts, web only): zlib inflates our output for the whole test-data zip corpus, and we inflate zlib's output in all its modes.
  • libvorbis comparison:
    • Short.ogg, Example.ogg and three default.sf3 samples (one per block size pair: 512/1024, 512/512, 256/2048) must have the exact libvorbis length and a maximum difference below 1e-5;
    • the *_libvorbis.pcm references were produced with libvorbisfile (float output);
    • they replace the bit-exact *_alphaTab.pcm references, which held the previous decoder's output including the end-of-packet defect.
  • IMDCT against its definition for all block sizes from 64 to 8192.
  • Short-paged.ogg (every packet split across pages, valid CRCs, decoded identically by libvorbis) decodes bit-identically to Short.ogg.
  • Corrupt Vorbis input: 300 deterministic mutations and truncations of Short.ogg either decode or throw FormatError.
  • Large-corpus check (local soundfonts, not committed): every stream of sonivox.sf3, GeneralUserGS.sf3, MS Basic.sf3, 32MbGM.sf3 and DSOUNDFONT Lite (~5400 streams) has the exact libvorbis length and a maximum difference of 6.6e-7.
  • Test suites: TypeScript 1972, C# 1968 and Kotlin 1968 tests pass.

Zip results (the 1,649 zip entries of the test data, 19.7 MB uncompressed):

Previous New
Inflate 459 ms 94 ms
Deflate 3,994 ms 274 ms
Compressed size 2,993,034 B 2,701,530 B (zlib at its default level: 2,715,566 B)

Vorbis speed (Node 24, decoding all samples of a soundfont):

Soundfont Audio Previous New
default.sf3 14.7 s 857 ms 34 ms
GeneralUserGS.sf3 363 s 6.5 s 0.66 s
MS Basic.sf3 5542 s not measured 8.2 s

Size (minified / gzip):

Module group Previous New
zip 22.4 K / 7.2 K 12.8 K / 4.6 K
Vorbis 33.5 K / 12.3 K 15.0 K / 7.0 K

Found along the way, not changed here

  • C# runtime: Int32Array.Set copies to the wrong offset when the destination is a subarray view. The new code doesn't use it.
  • GpifWriter: frets.sort() sorts numbers without a comparator. JS sorts them as text, C#/Kotlin numerically, so the platforms can write different output.
  • Transpiler:
    • a C# keyword used as @record field name (e.g. long) produces an unescaped named argument;
    • Kotlin keeps (a >> 2) + (b >> 2) as Int.

Checklist

  • I consent that this change becomes part of alphaTab under its current or any future open source license
  • This PR is linked to an accepted issue (see above)
  • Changes are implemented
  • New tests were added
  • I have read AGENTS.md if an AI helped draft any part of this PR

AI authorship disclosure

  • No AI agent authored any part of this PR (description, code, tests, or commit messages)
  • An AI agent contributed to this PR. The AI-authored disclosure block
    (alphatab-ai-authored-v1) is present at the top of this body, and I have
    personally reviewed every change and can explain each one

Further details

  • This is a breaking change
  • This change will require update of the documentation/website

…ct in C# and Kotlin

The CRC-32 table and state used unsigned 32-bit values and >>>. In the
transpiled C# and Kotlin code these hit out-of-range double-to-int
conversions and a signed shift, so exported zip entries carried wrong
checksums there. Add known-answer tests.
…lementation

Add DeflateFormat with the parts of the format shared by encoder and
decoder (length/distance tables, code length order, fixed code,
canonical code assignment) and its integer rules for the C# and Kotlin
builds.

The new Inflater decodes a whole stream into one growing buffer using
single-level lookup tables, reads input bytes only when needed so it
stops exactly at the end of the stream, and validates the data as
strictly as zlib. It replaces the Haxe based Inflate, HuffTools and
Huffman.
…lementation

The new Deflater compresses a whole buffer at once: hash chains with
lazy matching using zlib's default search limits, and per block the
smallest of stored, fixed and dynamic encoding. Dynamic codes are
length-limited Huffman codes computed with the Moffat-Katajainen
algorithm. All results are deterministic and identical on all
platforms.

ZipWriter now computes the CRC-32 itself. The SharpZipLib based
Deflater, DeflaterEngine, DeflaterHuffman, DeflaterConstants,
PendingBuffer and the unused Adler32 are removed.

Tests cover round trips, golden output, the code length limiting and,
on the web target, interoperability with zlib.
The zip inflate and deflate algorithms are no longer based on these
libraries.
Computes the Vorbis IMDCT through a DCT-IV based on an n/4 point complex FFT.
Covers all Vorbis block sizes (64-8192), including 64 and 128 where the
current implementation produces wrong output.
…ementation

Decodes the Ogg Vorbis samples of SF3 soundfonts with a decoder written from
the Vorbis I specification, replacing the NVorbis port:

- Codewords at the end of a packet are decoded when they fit into the
  remaining bits. They were dropped before, which left an error of up to
  -36 dBFS compared to libvorbis in SF3 samples.
- The output is cut at the final granule position in all streams.
- Packets spanning Ogg pages are read correctly.
- Parsed setups are cached per Hydra: SF3 soundfonts share a few setups
  across all samples.
- Floor type 0 (only used by encoders before 2002) is not supported.

The tests now compare against libvorbis output instead of the previous
decoder's output.
Short-paged.ogg is Short.ogg with every packet split across pages (valid CRCs,
decoded identically by libvorbis). Mutated and truncated streams must either
decode or fail with a FormatError, which guards the C# and Kotlin ports
against out-of-range access.
@Danielku15 Danielku15 self-assigned this Oct 11, 2026
@Danielku15
Danielku15 merged commit 96e7858 into develop Oct 11, 2026
6 checks passed
@Danielku15
Danielku15 deleted the refactor/vorbis-own-decoder branch October 11, 2026 16:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reimplement vendored code to reduce license coupling

1 participant