llamadart

llamadart is a Dart package for on-device inference. It runs GGUF models through llama.cpp and .litertlm bundles through LiteRT-LM, on Android, iOS, macOS, Linux, Windows and the web, behind one Dart API. Web support is experimental. Native targets need no network once the model is on the device.

• Streaming chat and text generation, with structured JSON output and cancellable task handles.
• Tool calling driven by the model's chat template.
• Embeddings, reranking, and image and audio input with multimodal models.
• Speech to text, text to speech, and decision models with cooperative task cancellation.
• Image generation from text prompts on native platforms (Preview).
• Runtime LoRA adapters for llama.cpp, observability hooks and typed capability checks.

Choose direct llamadart, Genkit, or Flutter Edge AI for your application. Genkit is a separate package; the Flutter Edge AI connector is owned upstream and pins its own llamadart version.

All five engine types implement LlamadartEngine<C extends EngineCapabilities> for shared capability inspection and disposal. Keep loading and requests on the concrete engine; see shared lifecycle ownership.

LFM2 Python-style tool calls use declared argument types and preserve copied string backslashes; see tool calling. The automatic grammar preserves copied singleton string spellings and can use an existing Python arguments wrapper for unambiguous JSON-escaped string values. Foreign XML closing tags remain raw diagnostics with no executable calls. Native llama.cpp applies schema-directed lazy grammar to LFM2 automatic Python tool calls. Web additionally requires bridge assets v0.1.82+ with a loaded-core prompt-penalty capability; older runtimes reject before inference. LFM2 replies cut off at a reported length limit expose raw partialToolCalls diagnostics separately from executable calls and history. LFM2 tool markers inside reasoning stay in reasoning and produce no executable calls. Explicit parallel-tool requests reach supported Dart tool grammars even when template probes do not recognize parallel history.

Install

dependencies:
  llamadart: ^0.17.5

dart pub get
# or
flutter pub get

Requirements: Dart SDK >=3.10.7; for Flutter apps, Flutter SDK >=3.38.0, iOS deployment target 16.4 and macOS deployment target 14.0 or newer. Consumers need no local C++ toolchain: the package build hook resolves native runtime archives on first build or run. Windows and other per-platform notes: Installation.

Flutter iOS and macOS apps that link the Apple XCFrameworks through Swift Package Manager add the companion packages for the runtimes they use. A companion shares its core's MAJOR.MINOR and constrains llamadart to it.

dependencies:
  llamadart: ^0.17.5
  llamadart_llama_cpp_flutter: ^0.17.3 # GGUF / llama.cpp
  llamadart_litert_lm_flutter: ^0.17.3 # Apple .litertlm / LiteRT-LM targets
  llamadart_stable_diffusion_flutter: ^0.17.3 # Apple image generation, opt-in

Pairing rules, the SwiftPM bootstrap and the privacy manifest are in Installation and Apple setup. Web apps run dart run llamadart:web_setup once; see the WebGPU bridge. Lazy tool-call grammars and custom grammar roots follow the loaded bridge core's lazyGrammar capability.

llamadart also ships agent skills that teach coding agents its APIs. Install them from your app's root:

dart run skills@ get

To check a desktop setup without a model, run dart run llamadart:doctor; see setup report scope.

First generation

import 'package:llamadart/llamadart.dart';

Future<void> main() async {
  final engine = await LlamaEngine.load(
    LlamaModel(
      ModelSource.parse(
        'hf://unsloth/SmolLM2-135M-Instruct-GGUF/'
        'SmolLM2-135M-Instruct-Q2_K.gguf',
      ),
    ),
  );

  try {
    final reply = await engine.create(
      const [
        LlamaChatMessage.fromText(
          role: LlamaChatRole.user,
          text: 'Explain local inference in one sentence.',
        ),
      ],
      params: const GenerationParams(maxTokens: 48),
    ).text();
    print(reply);
  } finally {
    await engine.dispose();
  }
}

Dispose every engine before a native host calls C exit(); see Model lifecycle. For multi-turn chat, wrap the engine in ChatSession: First chat session.

Runtimes and platforms

LlamaBackend() routes by model file type. After a model loads, engine.runtime names the runtime and await engine.capabilities reports what it supports, so branch on those instead of the file extension. Unsupported runtime and option combinations are rejected with LlamaUnsupportedException instead of silently degrading.

Model format	Typical use	Runtime
GGUF	Broad llama.cpp compatibility, Metal/Vulkan/CUDA/CPU, WebGPU bridge	llama.cpp
.litertlm	LiteRT-LM deployments, Android GPU/NPU-oriented bundles, Gemma-family LiteRT packages	LiteRT-LM

llama.cpp uses Metal on Apple platforms, Vulkan on Linux and Windows, and WebGPU in the browser. Android Vulkan is experimental and opt-in. LiteRT-LM can also use GPU and, on compatible Android deployments, NPU. Image generation is not available on the web. On iOS, use ImageModelPlacement(vaeOnCpu: true) to keep diffusion on Metal, or CPU generation. Metal VAE decoding needs a native context policy that preserves an intrinsic SDXS tiny decoder through model refresh. The pinned v0.3.0-2 runtime provides that policy; full VAE and external-decoder Metal decoding remain refused on iOS. This is not new physical-device qualification. Per-target support, device limitations and runtime options:

• Platform and backend support matrix, including known limitations
• Backend selection
• Runtime parameters
• Native runtime configuration

Current default runtime pins:

Runtime	Pin
Native llama.cpp / GGUF	leehack/llamadart-native@v0.8.0-2
Native LiteRT-LM / .litertlm	leehack/litert-lm-native@v0.18.1-4
Web llama.cpp / GGUF	leehack/llama-web-bridge-assets@v0.1.82
Web LiteRT-LM / .litertlm	@litert-lm/core@0.18.0

Native LiteRT-LM overrides must match the package's pre-1.0 C bindings. LiteRT-LM 0.19 / C API 1.0 is rejected before engine calls; use the pinned runtime until the consumer migration in #1677 is delivered.

Native overrides accept stable vMAJOR.MINOR.PATCH releases and explicit historical or nightly bNNNN artifacts. New nightly wrapper rebuilds use bNNNN-N; existing bNNNN-llamadart.N artifacts stay valid as consumption-only overrides. Nightly cores use canonical decimal spelling, and rebuild counters start at 1 without leading zeros. Build-hook overrides must always name an explicit tag; latest is limited to maintainer synchronization and header/binding regeneration. On Windows, runtime environment directory overrides select backend modules; use llamadart_native_path to replace the complete runtime. The full override rules are in Native runtime configuration.

Await engine disposal before host shutdown. Forced worker cleanup during native creation uses owner API v1 in the pinned v0.8.0-2 runtime. Raw allocation-to-registration gaps remain; see the shutdown contract.

Guides

Task	Docs
Resolve local paths, URLs, and Hugging Face sources	Finding models
Stream text and collect output	Generation and streaming
Generate typed JSON	Structured output
Use tool calling	Tool calling
Use images, audio, or projectors	Multimodal
Transcribe speech on device	Speech to text
Synthesize speech on device	Text to speech
Generate images on device (Preview)	Image generation
Answer typed questions with a decision model	Decision models
Generate embeddings and rerank	Embeddings
Load LoRA adapters	LoRA adapters
Save and restore KV state, write a custom backend	API levels
Trace and measure requests	Observability
Run Flutter Web / WebGPU	WebGPU bridge
Tune performance	Performance tuning
Upgrade from 0.16.x	0.17 migration guide

Examples

Run the Flutter chat app from a repository checkout, or try the hosted chat app.

• Basic Dart CLI
• Flutter chat app
• Laya Tetris, a Flutter game played by a decision model
• HTTP server example
• TUI coding agent example

Contributing

Repository checks, validation commands, and PR expectations are in AGENTS.md and the maintainer docs.

License

MIT. See LICENSE.