🛰️[v0.4.5] Release Note: LiliCorr Speculative Decoding, Soprano TTS & Multimodal Cache Optimization

[JamePeng]

[0.4.5] — LiliCorr Speculative Decoding, Soprano TTS & Multimodal Cache Optimization

Version 0.4.5 brings another substantial update to llama-cpp-python, focusing on speculative decoding performance, multimodal inference efficiency, TTS model compatibility, and runtime state reliability.

Given the scope of changes, this release advances directly to 0.4.5, incorporating the improvements developed during the 0.4.4 cycle.

Key additions include LiLiCorr speculative decoding, Soprano TTS support, improved multimodal media caching, more efficient MTP checkpoint management, and stronger consistency across sampling, state restoration, and cache reuse.

Highlights

1. LiLiCorr Speculative Decoding

• Introduced LiLiCorr block decoding with projected anchor handling and checkpoint recovery.
• Improved token suppression and candidate probability alignment for MTP and DFlash.
• Added native parity, rollback, confidence threshold, and interruption recovery tests.
• Validated with Nemotron-3.5-Bolt-74B-A9B-24GB.

RTX 5090 benchmark — ordinary decoding vs. LiLiCorr:

Metric	Ordinary	LiLiCorr	Improvement
Decode throughput	132.6 tok/s	198.0 tok/s	1.49×
Throughput including first-token latency	129.5 tok/s	180.1 tok/s	1.39×
Draft acceptance	—	33.4–59.1%	—

Benchmarked with full GPU offloading, greedy sampling, and a draft length of 7. Generated outputs differed between decoding modes.

2. Soprano TTS and Audio Stream APIs

• Added Soprano TTS support for non-streaming WAV and float32 PCM synthesis.
• Exposed audio stream row-count and flush-progress queries.
• Added model-specific validation, error recovery, documentation, and CLI examples.
• Verified Soprano-1.1-80M with BF16 weights and an F16 projector at 32 kHz on GPU.
• Confirmed byte-identical Qwen3-TTS and Pocket TTS outputs before and after the update under fixed-seed, same-GPU conditions.

3. Faster Multimodal Media Cache Reuse

• Unified media cache bookkeeping with collision-checked negative media IDs and native decoder positions.
• Improved reuse of image embeddings and media checkpoints across conversation turns.
• Introduced device-side media checkpoints when sequence-slot ownership permits, with safe fallback to host snapshots.
• Normalized BOS/EOS and Qwen vision boundaries to avoid duplicated tokens.

Multi-turn media cache improvements:

• Qwen3.5: media cache hit rates improved from 0% / 0% / 0% to 0% / 100% / 100% across three turns.
• Gemma4E: media cache hit rates improved from 0% / 0% / 100% to 0% / 100% / 100%.
• Qwen3.5 first-token latency decreased by approximately 60% on turns 2–3 in the measured multi-turn workload.
• Additional device-checkpoint optimization reduced Qwen3.5 first-turn latency by 7.4% in a separate controlled benchmark.

These results reflect the tested configurations; cache benefits vary with model architecture, prompt history, and output length.

4. MTP Checkpoint and Rollback Optimization

• Eliminated unnecessary final target-only checkpoints for completed speculative requests.
• Improved MTP verification callbacks, accepted-prefix handling, and native rollback validation.
• Reduced redundant feature copying and improved restoration safety.
• Strengthened recovery behavior for interrupted generation and invalid checkpoints.

Qwen3.5-9B-MTP Q8_0 benchmark on RTX 5090:

• Eliminated approximately 50 MiB of unnecessary state exports per request.
• Reduced checkpoint cleanup from 27–28 ms to approximately 0.1 ms.
• Improved end-to-end throughput by approximately 8–12% across two tested workloads.

5. State, Cache, and Sampling Reliability

• Improved hybrid prefix reuse when checkpoints are disabled.
• Strengthened state restoration and snapshot ownership validation.
• Unified longest-prefix matching across RAM, disk, and trie caches.
• Optimized trie maintenance with compressed edges and reduced metadata overhead.
• Preserved model-specific token suppression and grammar masks during sampling.
• Exposed LlamaModel.suppress_tokens() for accessing native suppressed token IDs.
• Added broader regression coverage for cache matching, sampler parity, rollback, and recovery.

6. Documentation, Testing, and API Updates

• Expanded documentation for LiLiCorr, MTP rollback, hybrid caching, and multimodal cache behavior.
• Added performance benchmarks covering cache reuse, speculative decoding, and checkpoint overhead.
• Consolidated regression tests to reduce redundant model loading and CI execution time.
• Fixed Python 3.9 CI compatibility by pinning affected jobs to Ubuntu 24.04.
• Synchronized llama.cpp / MTMD / GGML API bindings to 2026-10-24.

Looking Ahead

With version 0.4.5, the project continues moving beyond basic model compatibility toward more efficient, reliable, and reusable inference execution.

The combination of LiLiCorr speculative decoding, optimized MTP checkpoints, and improved multimodal cache reuse provides a stronger foundation for high-performance local inference, especially for large language models and multi-turn multimodal applications.

Future development will continue to focus on speculative decoding architectures, cross-request state reuse, multimodal execution efficiency, and broader model support.

---

Full Changelog: [Compare changes](ce261d1...b7cc3e1)

— JamePeng