Website · Documentation · Quickstart · Compatibility · Contributing
OrbitKV extends the KV cache of vLLM and SGLang beyond GPU memory. Store reusable prefixes in DRAM and optional SSD, then restore them after GPU eviction or an engine restart. This serves repeated documents, shared system prompts and long conversation prefixes.
Run an independent Cache Manager on each inference host. Engines own GPU pages and scheduling; Rust owns external replicas, byte budgets and transfer completion. The pinned official engines are vLLM 0.31.0 and SGLang 0.5.22. Start with Qwen3-8B dense TP=1/PP=1 DRAM or io_uring recovery and check the support matrix before extending the deployment. Version 0.1.0 is not yet published; interfaces may change before 1.0.
- Reuse beyond HBM: save prefixes in DRAM and optional SSD; restore after GPU eviction or an engine restart while the Manager stays alive.
- Share cache capacity: connect multiple local engines to one Manager or fetch compatible peer replicas through Mooncake TENT TCP/RDMA. Opt-in OrbitTE TCP and RDMA development profiles connect the existing peer grants and pinned owners; installed serving remains unqualified.
- Rust ownership: native cache and transfer owners manage byte budgets, reservations and completion; Python connects engine callbacks and layouts.
- Local discovery: bounded inventory streams maintain covered indexes; ordinary queries avoid per-key etcd calls and validate source grants.
See features for available paths and their support limits.
Optional storage codecs, reuse policies and CUDA copy kernels have separate qualification limits. Native P/D uses the engines' own transports and independent cache adapters; see P/D composition.
Follow the installation and quickstart to build a complete
orbitkv-llm wheel for your Python version and CUDA 13 runtime. Use a separate environment for each engine
and install the same OrbitKV build in the Manager environment. The Manager needs
compatible PyTorch and does not load model weights.
Start a Manager:
orbitkv-cache-manager --addr 127.0.0.1:50055 --http-addr 127.0.0.1:9091 --pool-size 8gbStart vLLM in its official release environment:
VLLM_USE_V2_MODEL_RUNNER=0 vllm serve /path/to/immutable-model \
--enforce-eager --enable-prefix-caching \
--kv-transfer-config '{"kv_connector":"OrbitKVConnector","kv_role":"kv_both","kv_connector_module_path":"orbitkv.vllm"}'For SGLang, follow its linker recipe. To verify an
external hit, keep the Manager alive, restart the engine and repeat a multi-block
prompt; check orbitkv_load_bytes_total. A repeated prompt in the same engine can
hit its native GPU cache instead.
| Task | Guide |
|---|---|
| Choose features and a model | Features, model recipes |
| Select an engine release and topology | Compatibility |
| Share capacity or add SSD/peers | Deployment, Manager configuration |
| Understand ownership and data paths | Architecture, transport |
| Measure latency and throughput | Performance, benchmark methodology |
| Develop and review changes | Contributing, completion plan |
Use the support matrix for experimental features and current qualification limits. Reviewed component speedups do not establish a serving advantage over native caching or LMCache.
Apache 2.0. Bundled dependencies retain their own licenses and notices.