🚀 [v0.4.3] Release Note: Multimodal Embeddings, MTP Image Speculation & Extended Batch / Audio Stream APIs

[JamePeng]

[0.4.3] — Multimodal Embeddings, MTP Image Speculation & Extended Batch APIs

Version 0.4.3 marks another significant step forward for llama-cpp-python, with major improvements to multimodal embeddings, speculative decoding, low-level batch processing, and runtime performance.

This release introduces dedicated MTMD multimodal embedding APIs for text, images, and audio, enables MTP speculative decoding for multimodal image chat, and expands LlamaBatchExt to support token, embedding, and hidden-state inputs. It also improves compatibility with models such as Qwen3.5/3.6/3.8-MTP for media, Qwen3-VL-Embedding, and EmbeddingGemma 2.

Beyond inference capabilities, the Python wheel build pipeline has been substantially optimized, reducing GitHub Actions build times from over 3–4 hours to approximately 1.5–2 hours.

Highlights

1. Multimodal Embedding Support

• Introduced MTMDEmbeddingHandler and LlamaEmbedding.embed_multimodal() for pooled text, image, and audio embeddings without chat templates.
• Support marker-based prompts and ordered text/media inputs, with explicit management of decoder positions and output ownership.
• Improved causal multimodal embedding support, preserving model attention modes and correct text/media positions.
• Fixed embedding output dimensions using n_embd_out, including projected embeddings such as EmbeddingGemma 2's 768-dimensional vectors.
• Improved numerical stability for Lp normalization and enforced non-causal microbatch limits.
• Added a dedicated Qwen3-VL embedding CLI profile, examples, and usage guides.

2. MTP Speculative Decoding for MTMD Image Chat

• Enabled MTP speculative decoding with multimodal image inputs, including media positions and shifted target hidden states.
• Improved synchronization between target and draft contexts, rollback recovery, and media-prefix replay.
• Fixed attention KV cleanup after partial device checkpoint restoration.
• Improved DFlash/DFlash2 draft filtering and speculative execution reliability.
• Added a GPU-based Qwen3.5 image-chat example with an MTP toggle, acceptance statistics, rollback metrics, and decode-time comparisons.
• Validated multimodal MTP execution with Qwen3.5-9B-MTP and its matching projector.

3. Extended Batch APIs and Runtime Reliability

• Introduced LlamaBatchExt with support for text tokens, embeddings, paired MTP inputs, and explicit batch ranges.
• Added hidden-state submission through llama_batch_ext_set_embd_state.
• Integrated extended batch processing with native llama_process encoding and decoding.
• Added synchronized graph reservation, input dimension validation, and clearer resource ownership.
• Improved context lifecycle management, LoRA adapter retention, error cleanup, and protection against double frees.

4. Faster Python Wheel Builds

• Reuse compiled native libraries across Python 3.10–3.14 wheels instead of rebuilding for every Python version.
• Build native CUDA libraries once per configuration and package compatible wheels in parallel.
• Validate native build manifests, library hashes, package versions, and library loading before publishing.
• Automatically propagate native CUDA versions into wheel packaging.

GitHub Actions build time: approximately 3–4+ hours → 1.5–2 hours.

5. Documentation, Testing & API Updates

• Expanded README and Wiki coverage for multimodal embeddings, batch APIs, MTP image decoding, attention modes, pooling, and resource ownership.
• Updated guidance for MoE caching, OpenVINO, logging, embedding normalization, and runtime configuration.
• Consolidated runtime and speculative decoding regression tests.
• Added real-model coverage for Qwen3-VL embeddings, MTP, DFlash, DFlash2, and EmbeddingGemma 2.
• Synchronized llama.cpp / MTMD / GGML API bindings to 2026-10-16.

Looking Ahead

With multimodal generation and understanding already established in the 0.4 series, version 0.4.3 extends these capabilities into multimodal embedding and accelerated inference with mtp.

The addition of MTMD embeddings and MTP image speculation lays the groundwork for more flexible multimodal applications, while the extended batch APIs provide a stronger foundation for future decoding techniques and model architectures.

Some execution paths remain model-dependent, but the project continues moving toward a more unified, reliable, and extensible Python runtime for llama.cpp.

---

Special thanks to @KLL535 for MTMD embedding testing and feedback in [Issue #184](#184).

Full Changelog:
v0.4.3 Changelog
[Compare 0.4.3 changes](8fe0a5f...5725112)

— JamePeng