diff --git a/README.md b/README.md index 3eb93335..877c255c 100644 --- a/README.md +++ b/README.md @@ -1,311 +1,189 @@
-![InferRSI β€” Faster inference. Verifiable progress.](docs/assets/inferrsi-banner.svg) +![InferRSI β€” an agent app to onboard and optimize models. The outer loop improves the engine and the inner loop improves the optimizer.](docs/assets/inferrsi-banner.svg) -**Turn inference bottlenecks into verified improvements.** +[πŸš€ Quickstart](#quickstart) Β· [πŸ” How It Works](#how-it-works) Β· [πŸ“Š Selected Results](#selected-results) Β· [πŸ“š Docs](#documentation) Β· [🀝 Contributing](#contributing) -[![Python 3.10+](https://img.shields.io/badge/Python-3.10%2B-3776AB?style=flat-square)](pyproject.toml) -[![CLI](https://img.shields.io/badge/Interface-CLI-14B8A6?style=flat-square)](docs/quickstart.md) -[![vLLM-Omni contributions](https://img.shields.io/badge/vLLM--Omni-Contributions-8B5CF6?style=flat-square)](#results) +
-[News](#news) Β· [Why InferRSI](#why-inferrsi) Β· [How It Works](#how-it-works) Β· [Results](#results) Β· [Quickstart](#quickstart) Β· [Architecture](#architecture) Β· [Docs](#docs) +InferRSI helps coding agents and engineers profile inference workloads, test +optimizations against a baseline, and verify output quality before adopting a +change. Bring an engine checkout, a workload, and your quality checks. - + -InferRSI is a **CLI workbench for inference optimization with coding agents**. Bring your benchmark and quality checks; compare implementations, trace every gain to its source, and carry useful tuning lessons into the next experiment. +## πŸš€ Quickstart - +**πŸ–₯️ Use the CLI.** Install from source and list the packaged benchmarks on a +development machine with Python 3.10+. -## πŸ“° News +```bash +git clone https://github.com/InferMatrixLab/InferRSI.git +cd InferRSI +python -m venv .venv +source .venv/bin/activate +python -m pip install . +inferrsi benchmarks list +``` -Selected release milestones; see [all releases](https://github.com/InferMatrixLab/InferRSI/releases) -for packages, checksums and the complete changelog. +From a target vLLM-Omni checkout, run the CPU readiness probe. -- **2026-11-14 β€” [v0.2.0](https://github.com/InferMatrixLab/InferRSI/releases/tag/v0.2.0):** - Formal release. Highlights since - v0.1.1 include three packaged optimization tasks (MoE forward, Qwen-Image - denoising and Qwen3-TTS), a unified optimization lifecycle, richer trajectory - and evidence dashboards, and expanded multimodal case studies. - [Release highlights](docs/release-notes/v0.2.0.md). Case studies retain their - workload-specific validation limits. -- **2026-11-01 β€” [v0.1.0 Developer Preview](https://github.com/InferMatrixLab/InferRSI/releases/tag/v0.1.0):** - First packaged preview release of the inference-optimization workbench. +```bash +inferrsi doctor --repo . --repo-type vllm_omni --backend cpu +``` - +Setup and task discovery run without starting a benchmark. For a target host, +select its backend (`cpu`, `cuda`, `rocm`, `ascend`, or `mlx`). +[Install guide](docs/quickstart.md#install-and-smoke) Β· [Release wheels and checksums](https://github.com/InferMatrixLab/InferRSI/releases) -## ⚑ Why InferRSI +**πŸ€– Use a coding agent.** Install the plugin and ask it to check the target +checkout, workload, and quality criteria before starting an experiment. +[Codex](docs/codex-plugin.md#install-from-the-release-zip) Β· [DeepSeek Harness](docs/dsh-plugin.md) Β· [WorkSwarm](docs/workswarm-plugin.md#download-and-install) -- **Profile first.** Ground each optimization in measured bottlenecks and sourced tuning knowledge. -- **Prove the gain.** Run repeated A/B comparisons with quality gates, source fingerprints, and an HTML evidence report. -- **Keep the lessons.** Search prior experience and export cited context for Codex or other coding agents. +**πŸ§ͺ Run your first experiment.** [Try a packaged benchmark](docs/optimization-benchmarks.md#start-the-complete-workflow) Β· [Bring your own workload](docs/quickstart.md#real-remote-experiment) -You drive profiling and code changes; InferRSI runs and evaluates the benchmark and validation commands you provide. + ## πŸ” How It Works -**Optimize with evidence. Reuse reviewed lessons.** Humans and coding agents -prepare candidates; Workbench runs controlled comparisons against the target -engine. Evidence guides the next experiment and may inform reusable knowledge, -including limitations from failed attempts. -[Architecture](#architecture) Β· [Roadmap](https://github.com/InferMatrixLab/InferRSI/issues/61). - - - - - InferRSI experiment and knowledge loops: humans or coding agents declare a task; Workbench compares an external target engine; evidence supports another experiment and optional reviewed lessons; candidate storage requires separate authorization - - - -Illustrative target workflow; no measured results. [Watch the MP4](docs/assets/inferrsi-loop.mp4). - -Knowledge retrieval is read-only; candidate storage is optional and separately -authorized. Automatic model-backed search and automatic knowledge promotion remain -planned. Missing or incomplete evidence stays `INCONCLUSIVE`. - -The **offline harness-improvement MVP** adds a separate Skill Eval path for supplied -declarative candidates. Its fixed-output experiments validate the controller; -real-model gains remain unmeasured. [Flow, limits and reproduction β†’](docs/harness-improvement.md) - - - -## 🧭 Choose your starting path - -| You want to… | Start at | -| --- | --- | -| Choose a starting point for a new model, performance, debug, accuracy, upgrade or engine task | Choose the [primary skill for the requested outcome](docs/knowledge-layout.md#seven-deliverables) and read its procedure with `inferrsi skills enter ` before the first evidence search. Follow its decision points using the actual run and acceptance criteria. | -| Modify InferRSI itself | [Contributor guide](docs/contributing.md) and [agent collaboration](docs/agent-collaboration.md), under an accepted Issue scope | -| Use the Workbench to run experiments | [Quickstart](docs/quickstart.md): prepared target environment, benchmark/quality contracts, authorized experiment, recorded evidence | -| Complete an optimization benchmark in the current Agent | [Iterative task sessions](docs/optimization-benchmarks.md), with frozen rounds, independent feedback and final retesting | -| Deliver an independent inference engine | [Engine delivery template](docs/template/README.md): copyable scaffold and recipe handoff; no InferRSI runtime dependency | - -For every CLI/Python operation β€” discovery, normalized reads, store candidates, experiments β€” see the [operation guide](docs/operations.md). +**Outer loop.** To improve the engine, the agent profiles the workload and +proposes a change. InferRSI checks quality and measures the candidate against +the baseline under the same frozen protocol. Round evidence guides the next +change. A fresh process retests the best candidate at the end of the campaign. + +**Inner loop.** To improve the optimizer, compare paired campaigns with +knowledge on and off to measure whether access to the library helps the agent. +Reviewed PRs add findings to the library for agents to consult in later +campaigns. +[Knowledge comparison](docs/benchmark-comparison.md) + +- **Knowledge** supplies source-linked skills, prior measurements, and implementation notes for the agent to consult. +- **Harness** freezes the experiment inputs and gates, runs the rounds, and records the verdict and evidence. +- **Benchmark** supplies packaged tasks, measurements, and acceptance rules. Project campaigns supply their own workload and checks. + +**Apps** assembles these capabilities into the CLI, client integrations, and +result views. +[Module responsibilities](docs/architecture.md#module-responsibilities-and-dependencies) Β· [Workflow diagram](docs/optimization-workflow-unification.md#既序图) Β· [Architecture decision #362](https://github.com/InferMatrixLab/InferRSI/issues/362#issuecomment-6722325612) Β· [Apps migration #371](https://github.com/InferMatrixLab/InferRSI/issues/371) + +**Review a run in the dashboard.** Compare baseline and candidate profiles and +trace each experiment back to its evidence. From a recorded run directory, +open it with `inferrsi dashboard --input .`, which writes `dashboard.html` beside the run. +[Dashboard guide](docs/dashboard.md#generate-and-open) Β· [Download the interactive example](docs/examples/dashboard-profile-review.html) + +

+ + InferRSI dashboard showing baseline and candidate profiling captures and links to experiment and knowledge traces + +
+ Dashboard demo with synthetic data. +

-## πŸ“Š InferRSI Γ— vLLM-Omni: Optimization Results - -InferRSI campaigns and upstream experiments cover image, video and speech models. -Each result applies to its linked workload and measured revisions. Some results -are reported directly by upstream contributors, with validation scoped to their benchmarks. - -
-Benchmarks, validation limits, and upstream PRs - -| Model | Hardware | Optimization approach | Measured gains | Validation and limits | Evidence | vLLM-Omni PR | -| --- | --- | --- | --- | --- | --- | --- | -| **Qwen-Image-2.1-Turbo** | 4Γ— B300 | | **384.16 β†’ 266.60 ms (βˆ’30.60%)**
Warm HTTP end-to-end latency, repeated prompts | Historical InferRSI campaign. Latencies are medians of five trial means from alternating AB/BA groups. All 44 RGBA images were pixel-identical. | [Campaign #3](https://github.com/InferMatrixLab/InferRSI/issues/3) Β· [Pinned results](docs/qwen-image-2.1-turbo-b300.md) | [#8739](https://github.com/vllm-project/vllm-omni/pull/8739) (merged) | -| **Qwen-Image-2.1-Turbo** | 1Γ— Ascend 910B1 | | **βˆ’22.20%** offline generation latency
Warm repeated prompts | Upstream parent-to-merge comparison. 18/18 RGBA images were pixel-identical. The 30% target was unmet. Both arms aborted during teardown, leaving lifecycle `INCONCLUSIVE`. | [Final-merge measurements](https://github.com/vllm-project/vllm-omni/pull/8729#:~:text=Final%2Dmerge%20performance%20and%20image%20comparison) Β· [Earlier Ascend cohorts](docs/qwen-image-2.1-turbo-ascend.md) | [#8729](https://github.com/vllm-project/vllm-omni/pull/8729) (merged) | -| **MiniMax-H3 Ref2VA** | 8Γ— Ascend 910B1 | | **407.27 β†’ 108.12 s (βˆ’73.45%)**
5 s video, HTTP end-to-end vs. dense inference | All 14 frozen quality comparisons passed. | [Campaign #293](https://github.com/InferMatrixLab/InferRSI/issues/293) Β· [Pinned results](evidence/h3-ref2va-ascend-293/results/final-5s/README.md) | [#9698](https://github.com/vllm-project/vllm-omni/pull/9698) (draft) | -| **MiniMax-H3 Ref2VA** | 4 logical Ascend 910C devices | | **291.25 β†’ 286.67 s (βˆ’1.57%)**
5 s video, HTTP end-to-end | One lossless screening pair. Strict quality checks passed for the screened candidate. Overall campaign `INCONCLUSIVE`. | [Campaign #297](https://github.com/InferMatrixLab/InferRSI/issues/297) Β· [Evidence PR #317](https://github.com/InferMatrixLab/InferRSI/pull/317) | [#9745](https://github.com/vllm-project/vllm-omni/pull/9745) (draft) | -| **MiniMax-H3-Image** | 4Γ— Ascend 910B1 | | **16.89 β†’ 10.82 s** Β· 4736Γ—3552 generation
**4.18 β†’ 2.67 s** Β· 1024Γ—1024 generation
**5.68 β†’ 4.57 s** Β· 1024Γ—1024 editing | Single-launch HTTP medians vs. the same-layout SP4 control. All 30 numerical image screens passed. Latency targets were unmet and final qualification remains `INCONCLUSIVE`. | [Campaign #390](https://github.com/InferMatrixLab/InferRSI/issues/390) Β· [Evidence PR #394](https://github.com/InferMatrixLab/InferRSI/pull/394) | [#10184](https://github.com/vllm-project/vllm-omni/pull/10184) (open) | -| **MiniMax-H3-Image** | 2Γ— RTX 3090 | | **36.62 β†’ 14.59 s (βˆ’60.16%)**
Warm mean HTTP delivery latency | One upstream A/B of complete deployment profiles. All nine image pairs had identical decoded RGB pixels, including warmups. Repeated qualification remains incomplete. | [Paired deployment benchmark](https://github.com/vllm-project/vllm-omni/pull/10243#:~:text=One%20paired%20deployment%20comparison) | [#10243](https://github.com/vllm-project/vllm-omni/pull/10243) (open) | -| **MiniCPM-o 4.5** | 1Γ— L20X | | **+15.55%** throughput Β· **βˆ’27.69%** median TTFA
Packed conditioning, C32, same 50-frame window
**+20.5%** at C16 Β· **+25.4%** at C32
TensorRT + MPS throughput | The packed-conditioning candidate missed the exploratory WER margin against the tuning reference at the 50-frame setting. The separate TensorRT + MPS comparison had speech-quality verdict `FAIL`. | [Campaign #49](https://github.com/InferMatrixLab/InferRSI/issues/49) | [#9218](https://github.com/vllm-project/vllm-omni/pull/9218), [#9188](https://github.com/vllm-project/vllm-omni/pull/9188) (both merged) | -| **Qwen3-Omni** | 2Γ— L20X | | **+7.79%** throughput Β· **βˆ’6.06%** mean RTF
One C64 pilot | Median time to first audio **regressed by 7.12%**. Campaign `INCONCLUSIVE`. | [Campaign #112](https://github.com/InferMatrixLab/InferRSI/issues/112) Β· [Pinned pilot](https://github.com/InferMatrixLab/InferRSI/tree/e1ae256fc075ea9ab64aec68da749cf4a8c22ac7/evidence/qwen3-omni-2gpu-20261023/results/pr9283-cudnn-performance) | [#9283](https://github.com/vllm-project/vllm-omni/pull/9283) (draft) | -| **LFM3-Omni-2B** | 1Γ— Ascend 910B1 | | **0.285 β†’ 0.593 requests/s**
Upstream offline Omni API A/B, C4, temperature 0 | Retained audio waveforms matched bit-for-bit within each tested temperature. | [Offline A/B benchmark](https://github.com/vllm-project/vllm-omni/pull/9827#:~:text=Performance%20results) | [#9827](https://github.com/vllm-project/vllm-omni/pull/9827) (merged) | -| **LingBot-World** | 4Γ— Ascend 910C1 | | **108.29 β†’ 103.89 s (βˆ’4.06%)**
Full-request latency, upstream AB/BA | Pre-codec tensors and decoded RGB frames matched exactly. | [KV gather benchmark](https://github.com/vllm-project/vllm-omni/pull/8112#:~:text=Performance%3A%20whole%2Dblock%20index_select%20KV%20gather%20on%20NPU) | [#8112](https://github.com/vllm-project/vllm-omni/pull/8112) (merged) | - -**Earlier Qwen-Image-2.1-Turbo Ascend measurements.** On one Ascend 910B1 at -1280Γ—704, BF16, eager, and eight native steps, the author reported a 3111.5 ms -baseline. Conditioning reuse with interleaved RoPE reached **2459.6 ms (βˆ’21.0%)** -with pixel-identical RGBA. Adding the opt-in VAE fastpath reached **2425.5 ms -(βˆ’22.0%)** and passed the reported quality gate with changed pixels. Each mean -covers three requests after two warmups. InferRSI verification covered the -import, so acceptance remains `INCONCLUSIVE`. The 30% target was unmet. The B300 -campaign and this cohort used different hardware and measurement protocols. -[Earlier Ascend report](docs/qwen-image-2.1-turbo-ascend.md) - -TTFA is time to first audio. RTF is real-time factor. WER is word error rate. - -Upstream PR statuses checked on 2026-11-12. - -
- - - -## πŸ§ͺ Standalone engine benchmarks - -### Standalone video/audio engines - -[h3-metal](https://github.com/zuiho-kai/h3-metal) provides Metal-native FastH3 and MiniMax-H3 text-to-video/audio inference on Apple GPUs. + -Author-reported results on **Apple M5 Max (40-core GPU), 128 GB, macOS 27**, at **768Γ—448, 49 frames, seed 0**, using real weights with one engine running at a time. +## πŸ“Š Selected Results -| Engine | Device | Steps | Cold | Warm | vs. MLX (cold / warm) | vs. h3.c (cold / warm) | -| --- | --- | --- | --- | --- | --- | --- | -| `PipeNetwork/minimax-h3-mlx` (4-bit) | Apple M5 Max (40-core GPU) | 20 | 153.1 s | 134.6 s | 1.00Γ— / 1.00Γ— | 0.44Γ— / 0.54Γ— | -| `antirez/h3.c`, default preset | Apple M5 Max (40-core GPU) | 20 | 67.6 s | 73.3 s | 2.26Γ— / 1.84Γ— | 1.00Γ— / 1.00Γ— | -| **h3-metal**, original H3 | Apple M5 Max (40-core GPU) | 20 | **54.29 s** | **29.89 s** | **2.82Γ— / 4.50Γ—** | **1.25Γ— / 2.45Γ—** | -| **h3-metal**, FastH3 4-step + block-sparse attention | Apple M5 Max (40-core GPU) | 4 ΒΉ | **48.23 s** | **18.20 s** | **3.17Γ— / 7.40Γ—** | **1.40Γ— / 4.03Γ—** | +Model bring-up and measured optimizations across text, image, video, and speech. +Performance tables highlight metrics that improved by at least 20% against the +listed baseline. The evidence column gives the scope and validation status. +πŸ”₯ marks the largest gain in its table, βœ… a row whose quality checks passed, and +⚠️ a failed or unqualified check, or an inconclusive campaign. -Cold includes the first clip's weight loading, text encoding and preparation; warm reuses loaded weights and prepared state. + -ΒΉ FastH3 uses a distilled 4-step checkpoint, unlike the 20-step configurations. Warm values average clips 2–4 for the 20-step rows and two A/B/B/A warm runs for FastH3. +### 🧩 Model onboarding -[Author's report and reproduction β†’](https://github.com/zuiho-kai/h3-metal/blob/53b7fa5d88e2497ab427ab9c7155535f7e6e6817/README.md#performance). These external results were not rerun in InferRSI. +| Model | Engine and hardware | Milestone | Evidence | +| --- | --- | --- | --- | +| **DeepSeek V4.1 Flash** | vLLM-Ascend Β· 8Γ— Ascend 910C | W8A8 service on the 8K-input / 1K-output workload. Six independent runs gave **39.28 / 44.07 / 56.86 ms/token** at C1 / C4 / C16; the 25 ms/token target remains unmet. | [Bring-up and baseline](https://github.com/InferMatrixLab/InferRSI/issues/353#issuecomment-6641425447) | +| **Qwen3.8-27B** | Standalone nano-vLLM-derived engine Β· 1Γ— H20 | FP8 service passed scoped HTTP/SSE lifecycle checks; quality and performance qualification incomplete. | [Service bring-up](https://github.com/InferMatrixLab/InferRSI/issues/57) Β· [Performance campaign](https://github.com/InferMatrixLab/InferRSI/issues/67) | +| **LFM3-Omni-2B** | vLLM-Omni Β· 1Γ— Ascend 910B1 | βœ… Two real-weight serving tests passed; a functional eager adapter served as the offline audio baseline. | [Serving checks and A/B benchmark](https://github.com/vllm-project/vllm-omni/pull/9827) | -### Standalone speech engines + -Concurrency 1 (C1) results reported by each project. Values are **engine / vLLM-Omni / SGLang-Omni**; compare within a row, since workloads and audio-frame definitions differ. +### πŸ’¬ Text serving -| Project | Device | Throughput (frames/s) ↑ | TTFA p50 (ms) ↓ | Report and status | -| --- | --- | --- | --- | --- | -| [Drama 3](https://github.com/InferMatrixLab/drama-3-single-b300-engine) | 1Γ— B300 | **309.17** / 195.89 / 169.80 | **32.83** / 47.29 / 79.38 | [3-run medians](https://github.com/InferMatrixLab/drama-3-single-b300-engine/blob/58a1882a4d941f8d21bfce6be37d3d5e7c9fde80/docs/current-results.md); overall target unmet | -| [Qwen3-TTS](https://github.com/InferMatrixLab/qwen3-tts-rsi-npu-engine) | 1Γ— Ascend 910B1 | **61.12** / 12.71 / 4.31 | **50.41** / 263.51 / 439.72 | [3-run means](https://github.com/InferMatrixLab/qwen3-tts-rsi-npu-engine/blob/a29f8193b557c415175d70919bdfb9bd3dcd51fc/evidence/campaign-20261017/same-card-performance.md); source campaign accepted | -| [MiniCPM-o 4.5](https://github.com/InferMatrixLab/minicpm-o-4.5-single-gpu-engine) | 1Γ— L20X | **397.43** / 187.52 / 278.72 | **126.2** / 295.3 / 381.2 | [5-run means](https://github.com/InferMatrixLab/minicpm-o-4.5-single-gpu-engine/blob/a4f4778f312791c2d190b48e521141887ed160c8/docs/current-results.md); target unmet, quality unqualified | +| Model | Measured change and baseline | Metric and hardware | Validation and evidence | +| --- | --- | --- | --- | +| **DeepSeek V4.1 Flash** | **+54.45% / +36.12% / +20.80%**
Baseline: 68,239 / 1,711.98 / 1,132.9 tokens/s per card | Per-card throughput, prefill / decode / end-to-end Β· Ascend A3, W4A8, 1P1D | Throughput and latency rechecked at each scenario's latency limit; decode rechecked once. Long-output accuracy not verified. [Workloads and evidence](https://github.com/jiaran-king/dsv4.1-optimize) | -MiniCPM's SGLang baseline returns a complete waveform, so its TTFA equals end-to-end latency. These external results were not rerun in InferRSI. + - - -## πŸš€ Quickstart - -Requires **Python 3.10+**. Licensed under [Apache-2.0](LICENSE). +### 🎨 Image and video generation -**InferRSI is a Developer Preview (pre-release), with no API stability promise.** -The [daily release bot](docs/releases.md) publishes a version when the default -branch changes. +| Model | Measured change and baseline | Metric and hardware | Validation and evidence | +| --- | --- | --- | --- | +| **MiniMax-H3 Ref2VA** | **βˆ’73.45%**
Baseline: Dense inference Β· 407.27 s | 5 s video, HTTP latency Β· 8Γ— Ascend 910B1 | **3.77Γ—** with lossy Cache-DiT. βœ… 14 / 14 quality comparisons passed. [Campaign](https://github.com/InferMatrixLab/InferRSI/issues/293) Β· [Benchmark](evidence/h3-ref2va-ascend-293/results/final-5s/README.md) | +| **MiniMax-H3 Ref2VA** | **βˆ’76.3%**
Baseline: In-run dense control Β· 701.95 s | 5 s video, 50-step denoising time Β· 4Γ— Ascend 950PR | **πŸ”₯ 4.22Γ—** in five retained A/B slots. ⚠️ Historical; performance and quality inconclusive. [Archive record](https://github.com/InferMatrixLab/InferRSI/issues/520) Β· [Recipe benchmark](https://github.com/vllm-project/vllm-omni/pull/9955#:~:text=in%2Drun%20dense%20median) | +| **MiniMax-H3-Image** | **βˆ’60.16%**
Baseline: Original layer offload Β· 36.62 s | Warm HTTP latency Β· 2Γ— RTX 3090 | One paired comparison; βœ… 9 pixel-identical outputs. [Upstream results](https://github.com/vllm-project/vllm-omni/pull/10243#:~:text=One%20paired%20deployment%20comparison) | +| **MiniMax-H3-Image** | **βˆ’35.9% / βˆ’36.1%**
Baseline: Same-layout SP4 control Β· 16.89 / 4.18 s | HTTP generation latency at 4736Γ—3552 / 1024Γ—1024 Β· 4Γ— Ascend 910B1 | βœ… 30 / 30 numerical checks passed. Latency targets unmet. [Benchmark](https://github.com/InferMatrixLab/InferRSI/pull/394) | +| **Qwen-Image-2.1-Turbo** | **βˆ’30.60%**
Baseline: Pinned control Β· 384.16 ms | Warm HTTP latency Β· 4Γ— B300 | βœ… 44 pixel-identical outputs. [InferRSI results](docs/qwen-image-2.1-turbo-b300.md) | +| **Qwen-Image-2.1-Turbo** | **βˆ’22.20%**
Baseline: Upstream parent Β· 2,921.23 ms | Warm offline generation latency Β· 1Γ— Ascend 910B1 | ⚠️ Upstream reference; 18 pixel-identical outputs. Teardown inconclusive. [Upstream results](https://github.com/vllm-project/vllm-omni/pull/8729#:~:text=Final%2Dmerge%20performance%20and%20image%20comparison) | -### Use InferRSI in Codex + -Install the plugin in two steps. Use a local Codex client with ZIP upload support, -Python 3.10+ and Git. +### πŸ”Š Speech and multimodal serving -1. **Download the package from GitHub.** Open - [Releases](https://github.com/InferMatrixLab/InferRSI/releases) and download - `inferrsi-codex-plugin-vX.Y.Z.zip` from the selected version's assets. -2. **Install it in Codex.** If your client offers **Plugins β†’ Add β†’ Upload plugin - archive**, use it to import the downloaded ZIP. Select **Install plugin** on - the InferRSI detail page if that action is available. Once installed, enable - InferRSI and start a new conversation in your target repository. +| Model | Measured change and baseline | Metric and hardware | Validation and evidence | +| --- | --- | --- | --- | +| **LFM3-Omni-2B** | **+108.1%**
Baseline: Functional eager adapter Β· 0.285 req/s | Offline greedy request throughput Β· 1Γ— Ascend 910B1, C4 | βœ… Retained audio bit-identical at every tested temperature. [Upstream results](https://github.com/vllm-project/vllm-omni/pull/9827#:~:text=Performance%20results) | +| **MiniCPM-o 4.5** | **+100.6% / βˆ’50.72%**
Baseline: Pinned upstream main Β· 11.46 req/s / 0.65548 RTF | Request throughput / mean real-time factor Β· 1Γ— L20X, C32 | ⚠️ Five matched runs per arm; speech-quality non-inferiority unestablished. [Campaign](https://github.com/InferMatrixLab/InferRSI/issues/431) Β· [Benchmark](https://github.com/vllm-project/vllm-omni/pull/10331) | +| **MiniCPM-o 4.5** | **βˆ’27.69%**
Baseline: Tuned 50-frame deployment before packed conditioning | Median time to first audio Β· 1Γ— L20X, C32 | ⚠️ Missed the exploratory word-error-rate margin. [Benchmark](https://github.com/vllm-project/vllm-omni/pull/9218) | +| **MiniCPM-o 4.5** | **+20.5% / +25.4%**
Baseline: Pinned upstream main before TensorRT + MPS | Throughput at C16 / C32 Β· 1Γ— L20X | ⚠️ Speech quality check failed. [Benchmark](https://github.com/vllm-project/vllm-omni/pull/9188) | +| **Qwen3-TTS 1.7B** | **+158%**
Baseline: Fresh no-PR control Β· 1.869 audio s/s | Streaming audio throughput Β· 1Γ— Ascend 910B1, C1 | **πŸ”₯** ⚠️ Historical upstream reference; Mandarin speaker similarity and some ASR scores regressed. [Upstream benchmark](https://github.com/vllm-project/vllm-omni/pull/9068) | +| **Drama 3** | **+31% / βˆ’23%**
Baseline: Piecewise graphs Β· 17.31 frames/s / 1.237 RTF | Throughput / mean real-time factor Β· 2Γ— Ascend 910B1, C1 | Two runs per arm; real-time-factor targets unmet. [Campaign](https://github.com/InferMatrixLab/InferRSI/issues/132) Β· [Benchmark](https://github.com/vllm-project/vllm-omni/pull/9308) | -[Installation video (English)](docs/assets/inferrsi-codex-install-en.mp4) + -The runtime and knowledge are bundled; core operations need no separate pip -install or InferRSI source checkout. The [plugin guide](docs/codex-plugin.md#install-from-the-release-zip) -covers checksum verification and client compatibility. The video highlights the -click order with English UI illustrations based on the supplied Codex screenshots; -it does not record a completed installation. + -Try a first task: +### βš™οΈ Standalone engines -> Use InferRSI to research this engine's memory bottleneck. Find relevant -> knowledge and explain what is needed for a controlled comparison. +Engine projects report these comparisons. Speech benchmarks use one concurrent +request per engine and measure audio frames per second and median time to first audio. -Or inspect existing evidence: +| Engine | Measured change and baseline | Metric and hardware | Benchmark scope and evidence | +| --- | --- | --- | --- | +| **Qwen3-TTS** | **+381% / βˆ’81%**
Baseline: vLLM-Omni Β· 12.71 frames/s / 263.51 ms | Audio throughput / time to first audio Β· 1Γ— Ascend 910B1 | **πŸ”₯** βœ… Three runs per engine; audio and contract checks passed. [Same-card benchmark](https://github.com/InferMatrixLab/qwen3-tts-rsi-npu-engine/blob/a29f8193b557c415175d70919bdfb9bd3dcd51fc/evidence/campaign-20261017/same-card-performance.md) | +| **MiniCPM-o 4.5** | **+112% / βˆ’57%**
Baseline: vLLM-Omni Β· 187.52 frames/s / 295.3 ms | Audio throughput / time to first audio Β· 1Γ— L20X | ⚠️ Target unmet; quality unqualified. [Same-card benchmark](https://github.com/InferMatrixLab/minicpm-o-4.5-single-gpu-engine/blob/a4f4778f312791c2d190b48e521141887ed160c8/docs/current-results.md) | +| **Drama 3** | **+58% / βˆ’31%**
Baseline: vLLM-Omni Β· 195.89 frames/s / 47.29 ms | Audio throughput / time to first audio Β· 1Γ— B300 | Three-run medians; overall target unmet. [Same-card benchmark](https://github.com/InferMatrixLab/drama-3-single-b300-engine/blob/58a1882a4d941f8d21bfce6be37d3d5e7c9fde80/docs/current-results.md) | +| **h3-metal** | **βˆ’59%**
Baseline: h3.c Β· 73.3 s | Warm video generation latency Β· Apple M5 Max | Both engines used the original MiniMax-H3 at 20 steps. [Author benchmark](https://github.com/zuiho-kai/h3-metal/blob/53b7fa5d88e2497ab427ab9c7155535f7e6e6817/README.md#performance) | -> Use InferRSI to inspect the existing run I provide and explain its verdict, -> missing evidence and next action. +[Run your first experiment](#quickstart) Β· [Benchmark details and historical records](docs/optimization-results.md) Β· [vLLM-Omni model optimization roadmap](https://github.com/InferMatrixLab/InferRSI/issues/416) -Setup checks readiness without starting benchmarks. Experiments run locally on -the prepared target machine using your accepted gates and budget. See the -[plugin guide](docs/codex-plugin.md#install-from-the-release-zip) for installation, -updates and troubleshooting. + -### Use InferRSI in other clients + -For DeepSeek Harness, build and install the [native DSH bundle](docs/dsh-plugin.md). -For openJiuwen WorkSwarm v0.2.7, use the [native plugin ZIP](docs/workswarm-plugin.md#download-and-install). +## πŸ“š Documentation -### Use the CLI +- **▢️ Run and review.** [Operations](docs/operations.md) Β· [Packaged optimization benchmarks](docs/optimization-benchmarks.md) Β· [Result delivery](docs/result-delivery.md) +- **🧠 Use knowledge.** [Skill library](src/inferrsi/knowledge/data/skill_library/README.md) Β· [CANNbot setup and revocation](docs/external-channels.md#setting-one-up) Β· [MindIE setup and revocation](docs/codex-plugin.md#enable-and-revoke) +- **🧭 Understand and extend.** [Module layout and import migration](docs/module-layout.md) Β· [Architecture and directory map](docs/architecture.md) Β· [Standalone engine template](docs/template/README.md) +- **πŸ“Œ Follow changes.** [Releases](https://github.com/InferMatrixLab/InferRSI/releases) Β· [Changelog](CHANGELOG.md) -This preview is not on PyPI. Download the wheel and `SHA256SUMS` from the selected -[Release](https://github.com/InferMatrixLab/InferRSI/releases), verify its checksum, -then install the downloaded wheel: + -```bash -python -m pip install "/inferrsi--py3-none-any.whl" -``` +## 🀝 Contributing -For CLI development from this checkout: +**🀝 People decide in issues; agents write the PRs.** -```bash -python -m venv .venv -source .venv/bin/activate -python -m pip install . +People agree on the scope, resources, and acceptance criteria in the issue. +Agents implement the agreed scope, run tests, and submit PRs with evidence. +To merge, a PR must pass the required checks and receive two independent +approvals. The issue owner uses the evidence to accept the work or reopen it. -# Find relevant tuning knowledge and follow the printed open command. -inferrsi find 'Qwen-Image-2.1 Ascend memory bottleneck' -# Export a context bundle for an agent that needs one. -inferrsi context --repo-type vllm_omni --backend cuda --output context.md -``` +Start with a reproducible workload, a baseline, and evidence of the bottleneck. +Sign off commits with `git commit -s`. +[Open an issue](https://github.com/InferMatrixLab/InferRSI/issues/new/choose) Β· [Human–agent collaboration](docs/agent-collaboration.md) Β· [Contributor guide](docs/contributing.md) Β· [License](LICENSE) -For Ascend image-generation context: +Run the local development checks before submitting a change. ```bash -inferrsi context --repo-type vllm_omni --backend ascend \ - --scenario diffusion.image_generation --output ascend-context.md +python -m pip install -e ".[dev]" +pytest +ruff check src tests ``` - -Ready to measure your own model? [Connect your benchmark and quality checks β†’](docs/quickstart.md#real-remote-experiment) - - - -## 🧩 Architecture - -**One experiment loop. Independent model, repository and hardware adapters.** -The implemented [Harness Γ— Model Γ— Infra boundary](docs/architecture.md) keeps new -campaigns out of the common AB/BA scheduler, process cleanup and evaluator. - -![InferRSI system architecture: inputs, experiment loop and evidence](docs/assets/system-architecture-en.svg) - -[English SVG](docs/assets/system-architecture-en.svg) Β· -[Chinese original](docs/assets/system-architecture-zh-CN.svg) - -The supplied overview and its English translation summarize the intended workflow. -Evidence feeds the next experiment through explicit review and user-selected -constraints; automatic evidence-to-knowledge promotion or RunSpec population is -not implemented. Process-cleanup behavior still needs platform-specific validation. -See the [source audit and proposed remedies](docs/reviews/source-audit-ac70272.md) -for the 34 review points, qualifications and suggested acceptance checks. - -Alongside the engine loop, the [offline improvement controller](docs/harness-improvement.md) -composes the existing Skill Eval comparator and independent verifier. Its acceptance -rules live in `benchmark/improvement.py` and its search progression in -`harness/improvement.py`; the tool adapter freezes snapshots and -carries the total budget across probes, rerun and holdout. The engine diagrams above -continue to describe the engine path; this additional lane does not provide engine -performance evidence or an autonomous model-backed optimizer. - -
-See the implemented modules - -![Implemented module architecture](docs/assets/modular-components-en.svg) - -
- -
-Find the right directory to extend - -![Module-to-directory map](docs/assets/modular-directory-map-en.svg) - -
- -[Extension guide and compatibility boundary β†’](docs/architecture.md#where-to-extend) Β· -[Original design diagrams (English / Chinese) β†’](docs/architecture.md#knowledge-and-evidence-ownership) - - - -## πŸ“š Go deeper - -| Start here | What you will find | -| --- | --- | -| [Quickstart](docs/quickstart.md) | Installation, real A/B experiments, and reports | -| [Operation guide](docs/operations.md) | Every CLI/Python operation: entry point, effect class, boundary and example | -| [Bounded skill evaluation](docs/skill-evaluation.md) | Real-client skill/task comparisons, independent outcomes and evidence limits | -| [Offline harness improvement](docs/harness-improvement.md) | One INV/DOM step, quality/cost selection, held-out gate and reproducible controller experiments | -| [Qwen B300](docs/qwen-image-2.1-turbo-b300.md) | Reproduction, profiling, optimization ledger and exact image checks | -| [Qwen Ascend](docs/qwen-image-2.1-turbo-ascend.md) | NPU results, rejected rounds and verification scope | -| [h3-metal lessons](docs/h3-metal-apple-silicon-lessons.md) | MiniMax-H3 on Apple silicon: measurement pitfalls, gains, rejected directions and search-loop lessons | -| [Apple silicon playbook](docs/apple-silicon-playbook.md) | Measured, engine-agnostic lessons for Metal/MLX engines: unified memory, Metal execution and benchmarking a shared Mac | -| [Knowledge base](src/inferrsi/knowledge/data/skill_library/records/README.md) | Three learning tiers: contribution, dependency impact and peer techniques | -| [Skill catalog](src/inferrsi/knowledge/data/skill_library/README.md) | Searchable records linking categorized skills to full packaged procedures and resources | -| [Source audit](docs/reviews/source-audit-ac70272.md) | 34 review points with proposed remedies, qualifications and validation criteria | -| [Daily learning prompt](docs/daily-learning-prompt.md) | 06:00 Asia/Shanghai L2C + performance workflow and prerequisites | -| [Complete skill library](src/inferrsi/knowledge/data/skill_library/README.md) | Task entrypoint, system knowledge, playbooks, workflows and casebooks | -| [vLLM-Omni experience records](src/inferrsi/knowledge/data/skill_library/records/README.md) | Backend-scoped tuning methods, sources, and applicability | -| [Contributor guide](docs/contributing.md) | Development workflow, evidence contracts, and CI | -| [Engine delivery template](docs/template/README.md) | Copyable engine scaffold, branding, evidence and schema-v1 catalog recipe | - -**Build with us.** Bring a bottleneck, a reproducible optimization, or a tuning lesson. [Open an issue β†’](https://github.com/InferMatrixLab/InferRSI/issues) Use `git commit -s` for contributions. diff --git a/docs/architecture.md b/docs/architecture.md index f76b1c11..f6fa8d4a 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -44,10 +44,9 @@ and [TTS service protocol](tts-benchmark.md) expose their documented execution boundaries. Apps and Harness callers assemble these operations into optimization rounds under the #362 target architecture. Each task records its qualification. -The local experiment path now has concrete Harness, Model and Infra modules. The -CLI, RunSpec, knowledge retrieval and existing command contracts remain stable. -The diagrams below describe implemented code, with the historical compatibility -boundary called out separately. +The diagrams below show where the CLI assembles an experiment, which package +owns each step and where to extend it. The [compatibility notes](#compatibility-and-historical-evidence) +cover retained interfaces and imports that moved. ## Three-objective delivery framing @@ -114,12 +113,9 @@ repository-pack description through the neutral `resources.repository_packs` leaf, under the one permanent importer rule in [module-layout.md](module-layout.md#permanent-neutral-leaf-importer-rule-proposed). -The implemented table above is not the target four-module layout. The -suggested rule, including what the import gate does not prove, is -[module-layout.md](module-layout.md). That page does not move code. It also -records the first-wave compatibility pairs that #412 R1 retired and the -permanent neutral-leaf importer rule, both held in -[legacy-knowledge-manifest.json](legacy-knowledge-manifest.json). +The [target module layout](module-layout.md) describes the migration and the +scope of the static import checks. The [migration manifest](legacy-knowledge-manifest.json) +records the retired compatibility pairs and the permanent neutral-leaf importer rule. The caller still prepares the benchmark runtime and baseline/candidate source checkouts. Separate working directories alone do not isolate Python imports, @@ -309,15 +305,16 @@ forwarding shell or alias: the seven Knowledge shells, whose code is in `reporting.py`, whose code is in `apps/legacy/`; and `inspection.py`, whose host binding is `apps/inspection.py`. The build-generated `_version.py` is ignored wheel metadata, not a tracked root file. -[module-layout.md](module-layout.md#retired-root-modules) maps each retired -path to its replacement and lists what still keeps the isolation gate false. +The [migration list](module-layout.md#retired-root-modules) maps each retired +path to its replacement. The [import gate](module-layout.md#import-gate) defines +the dependency checks. ## Where to extend | Add | Main extension locations | Required qualification | | --- | --- | --- | | Model or scenario | `harness/serving`, `benchmark/metrics.py`, `benchmark/validators`, `harness/scenarios` | Frozen request and metric contract; model-specific quality coverage | -| Repository | `apps/host/targets`, a composed scenario, `data/repo_packs` | Stable source identity; command/result contract; environment isolation | +| Repository | `apps/host/targets`, a composed scenario, `harness/data/repo_packs` | Stable source identity; command/result contract; environment isolation | | Hardware | `apps/host/platforms` | Device selection, process exclusivity and independent backend/model evidence; a backend is registered in `apps/host/platforms/__init__.py` and its probe command is defined in the implementing module under `apps/host/platforms/`, not in package data | The acceptance rule is: **a new model or hardware adapter must reuse the common diff --git a/docs/assets/inferrsi-banner.svg b/docs/assets/inferrsi-banner.svg index 26fca774..7a431dd3 100644 --- a/docs/assets/inferrsi-banner.svg +++ b/docs/assets/inferrsi-banner.svg @@ -1,38 +1,70 @@ - - - + + InferRSI β€” an agent app to onboard and optimize models + The outer loop improves the engine within one task. The agent proposes changes, InferRSI checks quality and performance, and each round guides the next until InferRSI retests the best candidate in a fresh process. The inner loop improves the optimizer across tasks. Paired runs with knowledge on and off evaluate the value of reuse, and reviewed pull requests can add findings to the library for future tasks. - - - - - - - - - - + + + + + + - - - - - INFERMATRIXLAB / INFERENCE SYSTEMS - InferRSI - Faster inference. Verifiable progress. - BENCHMARKS Β· QUALITY GATES Β· REUSABLE KNOWLEDGE - - - - - - - Profile - Optimize - Validate - Reuse - - Every experiment feeds the next. + + + INFERMATRIXLAB + InferRSI + An agent app to onboard and optimize models + + + + OUTER LOOP + Improve the engine within one task + INNER LOOP + Improve the optimizer across tasks + + + + + + + + Evidence for the next round + + + + + + + + + + + + Propose + Run & checkNΓ— + Keep best + Final retest + + + + + Knowledge library + Skills Β· cases Β· records + Evaluate reuse + Knowledge on vs. off + Run evidence + Patch Β· verdict Β· report + Library updates via reviewed PRs diff --git a/docs/assets/modular-components-en.svg b/docs/assets/modular-components-en.svg index fbe0d2da..d4a4f9d7 100644 --- a/docs/assets/modular-components-en.svg +++ b/docs/assets/modular-components-en.svg @@ -1,50 +1,47 @@ -InferRSI implemented module architectureThe entry layer composes Harness, Model and Infra. Shared modules are implemented; historical validators remain frozen. Automatic agents and distributed executors are future work. - - -INFERRSI / IMPLEMENTED ARCHITECTURE -One experiment loop. Independent extension boundaries. - -ENTRY + KNOWLEDGE -CLI β†’ immutable RunSpec β€’ Knowledge search β†’ cited manual context - - -HARNESS / ORCHESTRATION - -Runner + external scenario -AB/BA order and deadline -Source checks and trial evidence -execute / snapshot / metric hooks - -Media scenario -Warmups and seed ordering -First response + sample artifacts -Request callback; no model dispatch - -Evaluation + reporting -Performance and quality evidence -PASS / FAIL / INCONCLUSIVE -JSON records + HTML report - - -Scenario adapters compose these layers - -MODEL / CONTRACTS -Protocols + validators -Image payloads and metric interpretation -PNG/hash checks; RGB + alpha comparison -Campaign supplies model, settings and gates - -INFRA / EXECUTION -Targets + platforms + processes -Source identity; device discovery / qualification -Subprocess lifetime, HTTP timing and artifacts -No model constants or performance verdicts - -uses - -COMPATIBILITY BOUNDARY -Old CLI / import paths remain available. Historical LTX helpers and quality validators stay frozen. -New campaigns use shared modules and fresh helper hashes; old evidence keeps its original bundle. -Extend model / repository / hardware adapters without editing AB/BA, process cleanup or adjudication. -Future work: automatic planning and patching, SSH execution, distributed leases and Hub publication. + + How InferRSI assembles a benchmark + Apps connects Knowledge retrieval, Harness execution and Benchmark assessment. CLI flags and result formats survive the module moves. Python callers must update retired imports. + + + + + + + + How InferRSI assembles a benchmark + + + Apps + The CLI resolves the task and target. Workflows connect the inputs, execution and assessment. + + + + + + + + + Knowledge + Retrieves cited procedures, cases + and records when requested. + + + Harness + Executes trials, manages deadlines + and records what ran. + + + Benchmark + Supplies the packaged workload, + metrics and assessment rules. + + + Compatibility across module moves + CLI flags, external JSON metrics, run artifacts and reports are retained. Retired Python imports must be updated. diff --git a/docs/assets/modular-directory-map-en.svg b/docs/assets/modular-directory-map-en.svg index 908040f1..4d4113b8 100644 --- a/docs/assets/modular-directory-map-en.svg +++ b/docs/assets/modular-directory-map-en.svg @@ -1,58 +1,56 @@ -InferRSI module-to-directory mapImplemented Harness, Model and Infra modules map to package directories. Model, repository and hardware additions extend adapters while shared orchestration remains unchanged. - - -INFERRSI / WHERE TO EXTEND -A directory map for the next campaign. -Package root: src/inferrsi/ β€’ All solid panels below contain implemented modules. - -harness/ -Experiment composition -runner.py AB/BA + injected ports -scenarios/external.py Command trials -Sampling -> benchmark/sampling.py -Gates -> benchmark/legacy_run_v0.py -Old run, report -> apps/legacy/ -api.py, local_process.py Execute + cleanup - -model/ -Request and quality contracts -protocols/image.py Image API + PNG response -protocols/metrics.py Metric schema checks -validators/image.py Hashes + RGB/alpha gates - -Model pins and thresholds: campaign-owned - -infra/ -Environment and resource ownership -targets/git.py Source identity -platforms/ Probes + qualification -executors/local.py Subprocess + deadline -processes.py Owned worker cleanup -http.py, artifacts.py Transport + records -inspection.py, environment.py, commands.py -campaign.py Utility import facade - -Entry + data/ -Existing contracts and reusable knowledge -cli.py, arguments.py CLI entry -runspec.py Immutable RunSpec -catalog.py Search + cited context -skill_library/records/ Findings -skill_library/config/ Discovery config - -examples/ Campaign composition -tests/fixtures/legacy/ Frozen compatibility tests - -ADD A MODEL -ADD A REPOSITORY -ADD HARDWARE -model/ + harness/scenarios/ -Freeze protocol and quality checks -infra/targets/ + repo_packs/ -Compose the benchmark scenario -infra/platforms/ (probe commands) -Qualify the backend independently - -PRESERVED COMPATIBILITY -Root execution / inspection / evaluation / reporting wrappers; historical validators and their LTX helper. + + Selected directories under src/inferrsi/ + Apps owns the CLI, host bindings and workflows. Harness runs trials, Benchmark defines and assesses tasks, and Knowledge owns retrieval and the skill library. RunSpec and the repository pack reader have separate paths. + + + + + Where to extend InferRSI + Selected paths under src/inferrsi/ + + + apps/ + cli/ + Command entry and arguments + host/ + Repository and device probes + workflows/ + Application assembly + + + harness/ + runner.py + AB/BA trials + local_process.py + Deadlines and cleanup + records/ + Campaign records and traces + + + benchmark/ + tasks/ + Packaged workloads + metrics.py + Metric contract + validators/ + Output quality checks + + + knowledge/ + catalog.py + Search and cited context + data/skill_library/ + Procedures and findings + + contracts/runspec.py + Immutable experiment specification + resources/repository_packs.py + Shared reader for repository descriptions diff --git a/docs/contracts/current-design.md b/docs/contracts/current-design.md index 50b30e58..b99dd9b3 100644 --- a/docs/contracts/current-design.md +++ b/docs/contracts/current-design.md @@ -25,14 +25,14 @@ Historical candidate text, review dispositions, and the earlier Proposed drafts ## D001 β€” One distribution, three cores plus Apps, document contracts rather than shared business DTOs -Decision: keep Knowledge, Harness, and Benchmark as sibling independent cores; Apps is the only assembler. Each core owns its types. Cross-boundary traffic uses versioned JSON envelopes with explicit conversion; there is no shared business `contracts` package. The technical-only `inferrsi.contracts` package re-exports only `RunSpec`, defined once in `contracts/runspec.py`; `contracts/claim.py` owns the same Verdict enum re-exported by Harness. These are not shared business DTOs. The old root `contracts.py`, `runspec.py`, `claim.py`, and `_json_v1.py` files are retired without forwarding shells, and so are the old root `_repo_pack_data.py` and `evidence_paths.py`. R1 (#412) retired eleven more root modules the same way: the seven Knowledge compatibility shells, the three old-run forwarders `evaluation.py`, `execution.py` and `reporting.py`, and root `inspection.py`, whose host binding is now `inferrsi.apps.inspection` over the unchanged `inferrsi.infra.inspection` facts API. [`CHANGELOG.md`](../../CHANGELOG.md) maps every retired path to its replacement and states the import and pickle impact. +Decision: keep Knowledge, Harness, and Benchmark as sibling independent cores; Apps is the only assembler. Each core owns its types. Cross-boundary traffic uses versioned JSON envelopes with explicit conversion; there is no shared business `contracts` package. The technical-only `inferrsi.contracts` package re-exports only `RunSpec`, defined once in `contracts/runspec.py`; `contracts/claim.py` owns the same Verdict enum re-exported by Harness. These are not shared business DTOs. The old root `contracts.py`, `runspec.py`, `claim.py`, and `_json_v1.py` files are retired without forwarding shells, and so are the old root `_repo_pack_data.py` and `evidence_paths.py`. R1 (#412) retired eleven more root modules the same way: the seven Knowledge compatibility shells, the three old-run forwarders `evaluation.py`, `execution.py` and `reporting.py`, and root `inspection.py`, whose host binding is now `inferrsi.apps.inspection` over the facts API in `inferrsi.apps.host.inspection`. [`CHANGELOG.md`](../../CHANGELOG.md) maps every retired path to its replacement and states the import and pickle impact. Permitted shared technical leaves are only: - `inferrsi.contracts.json_v1` β€” the canonical encoder (D002); - `inferrsi.contracts.runspec` and `inferrsi.contracts.claim` β€” the existing RunSpec and Verdict technical types, with only RunSpec re-exported by the package initializer; - `inferrsi.contracts.evidence` β€” the existing evidence segment names and pure path rules; it reads and writes no filesystem state and is not re-exported by the package initializer; -- `inferrsi.resources.repository_packs` β€” the neutral raw pack-data reader and the existing static data enumeration (D020), formerly root `inferrsi._repo_pack_data`. Its parent initializer `resources/__init__.py` imports nothing. Outside Knowledge it has one permanent importer, `inferrsi.infra.inspection`, which binds only `repository_pack`; that exact rule is not a compatibility pair. A1 (#408) removed the separate temporary entry for root `arguments.py`: the moved parser `inferrsi.apps.cli.arguments` reads the same `REPO_TYPES` object through `inferrsi.knowledge.catalog` and does not import the leaf. +- `inferrsi.resources.repository_packs` β€” the neutral raw pack-data reader and the existing static data enumeration (D020), formerly root `inferrsi._repo_pack_data`. Its parent initializer `resources/__init__.py` imports nothing. Outside Knowledge it has one permanent importer, `inferrsi.apps.host.inspection`, which binds only `repository_pack`; that exact rule is not a compatibility pair. A1 (#408) removed the separate temporary entry for root `arguments.py`: the moved parser `inferrsi.apps.cli.arguments` reads the same `REPO_TYPES` object through `inferrsi.knowledge.catalog` and does not import the leaf. The neutral leaf carries no target selection, task, scoring, scheduling, or public business DTO, and imports none of K/H/B/Apps. It still finds the Harness-owned `pack.json` through `files("inferrsi").joinpath("harness", "data", "repo_packs", ...)`, so reading it never imports Harness. Creating the evidence directories is host I/O and stays in Apps, in `inferrsi.apps.storage.evidence_layout`; that module is not a shared leaf. The list is exact: a `contracts.` or `resources.` prefix admits no other module. diff --git a/docs/contributing.md b/docs/contributing.md index 636fbde4..59cf5c77 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -36,7 +36,7 @@ Nightly checks cover Linux and Windows checkouts with `core.autocrlf` set to `false`, `true` and `input`. Use `harness/scenarios`, `harness/serving` and `benchmark/validators` for new models, -`apps/host/targets` with `data/repo_packs` for repositories, and +`apps/host/targets` with `harness/data/repo_packs` for repositories, and `apps/host/platforms` for hardware. A backend is registered in `apps/host/platforms/__init__.py`; its probe command is defined in the implementing module under `apps/host/platforms/`. That diff --git a/docs/optimization-results.md b/docs/optimization-results.md new file mode 100644 index 00000000..0a30ce97 --- /dev/null +++ b/docs/optimization-results.md @@ -0,0 +1,140 @@ +# Optimization results + +[Back to InferRSI](../README.md#selected-results) Β· [vLLM-Omni model optimization roadmap](https://github.com/InferMatrixLab/InferRSI/issues/416) + +The README highlights measured improvements of at least 20%. This page keeps the +full curated record, including the campaigns below that bar. Hardware, workloads +and baseline revisions differ across comparisons. Quality and acceptance status +are reported for each result. + +## Text serving + +### DeepSeek V4.1 Flash W4A8 on Ascend A3 + +The project ran DeepSeek V4.1 Flash 1017 W4A8 on an Ascend A3 SuperPod with +disaggregated prefill and decode (1P1D, 8 physical cards per side). AISBench +generated load and InferRSI rechecked throughput at each scenario's latency +limit. [Project report, configuration and reproduction](https://github.com/jiaran-king/dsv4.1-optimize). + +| Scenario | Per-card throughput metric | Latency limit | Baseline β†’ optimized (tokens/s) | Measured change | +| --- | --- | --- | --- | --- | +| 19 Β· Prefill | Input throughput | Mean TTFT < 10 s | 68,239 β†’ 105,397 | **+54.45%** | +| 21 Β· Decode, C1152 | Steady-state output | Mean TPOT < 50 ms | 1,711.98 β†’ 2,330.28 | **+36.12%** | +| 23 Β· End-to-end, steady state | Decode output | Mean TPOT < 21 ms | 1,132.9 β†’ 1,368.5 | **+20.80%** | + +Cases 19 and 23 use medians from two runs. Case 21 uses one recheck of the +retained configuration, which includes a SiLU safeguard. TTFT is time to first +token. TPOT is time per output token. + +### Historical DeepSeek V4.1 Flash W8A8 pilot + +The A3 / 910C pilot used 16 dies, TP4 / DP4 / EP16, DSpark-5 and an +8K-input / 1K-output workload. The slide deck reported the following single-run +observations against the earlier control in [issue #329](https://github.com/InferMatrixLab/InferRSI/issues/329). + +| Concurrency | Earlier control β†’ reported pilot TPOT (ms/token) | Measured change | +| --- | --- | --- | +| 1 | 93.86 β†’ 30.02 | **βˆ’68.0%** | +| 4 | 97.36 β†’ 24.98 | **βˆ’74.3%** | +| 16 | 102.33 β†’ 36.55 | **βˆ’64.3%** | + +Only C4 reached 25 ms/token in the reported run. The follow-up benchmark uses +its own calibrated baseline. The campaign stopped with an **INCONCLUSIVE** +verdict. [Follow-up #353](https://github.com/InferMatrixLab/InferRSI/issues/353). + +## Image, video and speech in vLLM-Omni + +InferRSI campaigns and upstream experiments cover image, video and speech models. +Each result applies to its linked workload and measured revisions. Some results +are reported directly by upstream contributors, with validation scoped to their benchmarks. + +| Model | Hardware | Optimization approach | Measured gains | Validation and limits | Evidence | vLLM-Omni PR | +| --- | --- | --- | --- | --- | --- | --- | +| **Qwen-Image-2.1-Turbo** | 4Γ— B300 |
  • Text-conditioning reuse
  • BF16 operator fusion
  • Symmetric-memory Ulysses with batched QKV exchange
| **384.16 β†’ 266.60 ms (βˆ’30.60%)**
Warm HTTP end-to-end latency, repeated prompts | Historical InferRSI campaign. Latencies are medians of five trial means from alternating AB/BA groups. All 44 RGBA images were pixel-identical. | [Campaign #3](https://github.com/InferMatrixLab/InferRSI/issues/3) Β· [Pinned results](qwen-image-2.1-turbo-b300.md) | [#8739](https://github.com/vllm-project/vllm-omni/pull/8739) (merged) | +| **Qwen-Image-2.1-Turbo** | 1Γ— Ascend 910B1 |
  • Conditioning cache for repeated prompts
  • Shared modulation once per denoising step
  • Native interleaved NPU RoPE with frequency-table reuse
| **2,921.23 β†’ 2,272.83 ms (βˆ’22.20%)** offline generation latency
Warm repeated prompts | Upstream parent-to-merge reference. InferRSI imported the techniques as knowledge. 18/18 RGBA images were pixel-identical. The 30% target was unmet. Both arms aborted during teardown, leaving lifecycle `INCONCLUSIVE`. | [Final-merge measurements](https://github.com/vllm-project/vllm-omni/pull/8729#:~:text=Final%2Dmerge%20performance%20and%20image%20comparison) Β· [Earlier Ascend cohorts](qwen-image-2.1-turbo-ascend.md) | [#8729](https://github.com/vllm-project/vllm-omni/pull/8729) (merged) | +| **MiniMax-H3 Ref2VA** | 8Γ— Ascend 910B1 |
  • Lossy Cache-DiT profile tuning
| **407.27 β†’ 108.12 s (βˆ’73.45%)**
5 s video, HTTP end-to-end vs. dense inference | All 14 frozen quality comparisons passed. | [Campaign #293](https://github.com/InferMatrixLab/InferRSI/issues/293) Β· [Pinned results](../evidence/h3-ref2va-ascend-293/results/final-5s/README.md) | [#9698](https://github.com/vllm-project/vllm-omni/pull/9698) (draft) | +| **MiniMax-H3 Ref2VA** | 4Γ— Ascend 950PR (~86 GB per card) |
  • USP4 + MXFP8
  • RainFusion + Cache-DiT
| **701.95 β†’ 166.17 s (βˆ’76.3%, 4.22Γ—)**
5 s clip denoising time vs. the in-run dense control | Five retained historical A/B slots. Performance and quality qualification remained inconclusive. For quality, the recipe reports sharpness, flicker, and luma ratios of 0.924 / 0.950 / 0.984 against a dense USP4 + MXFP8 reference. | [Archive record #520](https://github.com/InferMatrixLab/InferRSI/issues/520) Β· [Measured repeats](https://github.com/vllm-project/vllm-omni/pull/9955#:~:text=in%2Drun%20dense%20median) | [#9955](https://github.com/vllm-project/vllm-omni/pull/9955) (merged recipe) | +| **MiniMax-H3-Image** | 4Γ— Ascend 910B1 |
  • Opt-in VAE tile graphs
  • Approximate encoder/decoder tile batching
| **16.89 β†’ 10.82 s (βˆ’35.9%)** Β· 4736Γ—3552 generation
**4.18 β†’ 2.67 s (βˆ’36.1%)** Β· 1024Γ—1024 generation | Single-launch HTTP medians vs. the same-layout SP4 control. All 30 numerical image screens passed across generation and editing. Latency targets were unmet and final qualification remains `INCONCLUSIVE`. | [Campaign #390](https://github.com/InferMatrixLab/InferRSI/issues/390) Β· [Evidence PR #394](https://github.com/InferMatrixLab/InferRSI/pull/394) | [#10184](https://github.com/vllm-project/vllm-omni/pull/10184) (merged) | +| **MiniMax-H3-Image** | 2Γ— RTX 3090 |
  • Rank-local layerwise offload
  • Selected DiT weights kept resident
  • AdaLN caching
  • Request-local text refinement reuse
| **36.62 β†’ 14.59 s (βˆ’60.16%)**
Warm mean HTTP delivery latency | One upstream A/B of complete deployment profiles. All nine image pairs had identical decoded RGB pixels, including warmups. Repeated qualification remains incomplete. | [Paired deployment benchmark](https://github.com/vllm-project/vllm-omni/pull/10243#:~:text=One%20paired%20deployment%20comparison) | [#10243](https://github.com/vllm-project/vllm-omni/pull/10243) (open) | +| **MiniCPM-o 4.5** | 1Γ— L20X | Seed-TTS runtime and deployment changes | **11.46 β†’ 22.99 requests/s (+100.6%)** and **0.65548 β†’ 0.32300 mean RTF (βˆ’50.72%)** at C32
**10.62 β†’ 18.51 requests/s (+74.29%)** and **0.34287 β†’ 0.19744 mean RTF (βˆ’42.42%)** at C16
**2.94 β†’ 3.66 requests/s (+24.49%)** at C1 | Five retained fresh servers per arm, 12,800 successful requests. The independent-engine target was outside this comparison. Formal speech-quality non-inferiority remains unestablished. | [Campaign #431](https://github.com/InferMatrixLab/InferRSI/issues/431) | [#10331](https://github.com/vllm-project/vllm-omni/pull/10331) (merged) | +| **MiniCPM-o 4.5** | 1Γ— L20X |
  • Packed conditioning on a tuned configuration
  • TensorRT speaker extraction + CUDA MPS (separate comparison)
| **βˆ’27.69%** median TTFA
Packed conditioning, C32, same 50-frame window
**+20.5%** at C16 Β· **+25.4%** at C32
TensorRT + MPS throughput | The packed-conditioning candidate missed the exploratory WER margin against the tuning reference at the 50-frame setting. The separate TensorRT + MPS comparison had speech-quality verdict `FAIL`. | [Campaign #49](https://github.com/InferMatrixLab/InferRSI/issues/49) | [#9218](https://github.com/vllm-project/vllm-omni/pull/9218), [#9188](https://github.com/vllm-project/vllm-omni/pull/9188) (both merged) | +| **LFM3-Omni-2B** | 1Γ— Ascend 910B1 |
  • Depth graphs
  • Grouped greedy sampling
| **0.285 β†’ 0.593 requests/s (+108.1%)**
Computed from the rounded means; upstream quotes +108.4% from its unrounded values.
Upstream offline Omni API A/B, C4, temperature 0 | Retained audio waveforms matched bit-for-bit within each tested temperature. | [Offline A/B benchmark](https://github.com/vllm-project/vllm-omni/pull/9827#:~:text=Performance%20results) | [#9827](https://github.com/vllm-project/vllm-omni/pull/9827) (merged) | +| **Drama 3** | 2Γ— Ascend 910B1 | Full graph replay for steady-state decode | **17.31 β†’ 22.40 / 22.91 frames/s (+31%)** and **1.237 β†’ 0.962 / 0.947 mean RTF (βˆ’23%)** at C1 | Rounded changes use the mean of the two candidate runs. The real-time factor targets were unmet. | [Campaign #132](https://github.com/InferMatrixLab/InferRSI/issues/132) | [#9308](https://github.com/vllm-project/vllm-omni/pull/9308) (merged) | +| **Qwen3-TTS 1.7B** | 1Γ— Ascend 910B1 | Codec and predictor optimizations | **1.869 β†’ 4.822 audio s/s (+158%)** at C1
**16.463 β†’ 38.755 (+135%)** at C16
**21.245 β†’ 52.091 (+145%)** at C32 | Historical seven-path profile vs. fresh same-card no-PR control. Mandarin speaker similarity and some ASR results regressed. Upstream reference. | [vLLM-Omni roadmap #416](https://github.com/InferMatrixLab/InferRSI/issues/416) | [#9068](https://github.com/vllm-project/vllm-omni/pull/9068) (merged) | + +**Earlier Qwen-Image-2.1-Turbo Ascend measurements.** On one Ascend 910B1 at +1280Γ—704, BF16, eager, and eight native steps, the author reported a 3111.5 ms +baseline. Conditioning reuse with interleaved RoPE reached **2459.6 ms (βˆ’21.0%)** +with pixel-identical RGBA. Adding the opt-in VAE fastpath reached **2425.5 ms +(βˆ’22.0%)** and passed the reported quality gate with changed pixels. Each mean +covers three requests after two warmups. InferRSI verification covered the +import, so acceptance remains `INCONCLUSIVE`. The 30% target was unmet. The B300 +campaign and this cohort used different hardware and measurement protocols. +[Earlier Ascend report](qwen-image-2.1-turbo-ascend.md) + +### Historical campaigns below the 20% highlight bar + +These campaigns stay on the record even though their measured gain is under the +bar the README highlights. Their upstream PR labels are kept as they were. + +| Model | Hardware | Measured change | Validation and limits | Evidence | vLLM-Omni PR | +| --- | --- | --- | --- | --- | --- | +| **MiniMax-H3 Ref2VA** | 4 logical Ascend 910C devices | **291.25 β†’ 286.67 s (βˆ’1.57%)**
5 s video, HTTP end-to-end | One lossless screening pair; strict quality checks passed for the screened candidate. Campaign `INCONCLUSIVE`. | [Campaign #297](https://github.com/InferMatrixLab/InferRSI/issues/297) Β· [Evidence PR #317](https://github.com/InferMatrixLab/InferRSI/pull/317) | [#9745](https://github.com/vllm-project/vllm-omni/pull/9745) (draft) | +| **Qwen3-Omni** | 2Γ— L20X | **+7.79%** throughput Β· **βˆ’6.06%** mean RTF
One C64 pilot | Median time to first audio regressed 7.12%. Campaign `INCONCLUSIVE`. | [Campaign #112](https://github.com/InferMatrixLab/InferRSI/issues/112) Β· [Pinned pilot](https://github.com/InferMatrixLab/InferRSI/tree/e1ae256fc075ea9ab64aec68da749cf4a8c22ac7/evidence/qwen3-omni-2gpu-20261023/results/pr9283-cudnn-performance) | [#9283](https://github.com/vllm-project/vllm-omni/pull/9283) (draft) | +| **LingBot-World** | 4Γ— Ascend 910C1 | **108.29 β†’ 103.89 s (βˆ’4.06%)**
Full-request latency, upstream AB/BA | Pre-codec tensors and decoded RGB frames matched exactly. | [KV gather benchmark](https://github.com/vllm-project/vllm-omni/pull/8112#:~:text=Performance%3A%20whole%2Dblock%20index_select%20KV%20gather%20on%20NPU) | [#8112](https://github.com/vllm-project/vllm-omni/pull/8112) (merged) | + +### Historical MiniMax-H3 Ref2VA step-reduction measurements on 950PR + +The author reported these 5-second workloads on four Ascend 950PR cards +relative to their corresponding 50-step dense controls. Each sampling +configuration used three seeds and eleven prompts. Both performance and quality +qualification remained **INCONCLUSIVE**. One seed in the 10-step dense +configuration lost 9.17% subject consistency. The 10-step Cache-DiT route +combines fewer denoising steps and lossy caching, so its speedup describes the +whole workload change. [Record #520](https://github.com/InferMatrixLab/InferRSI/issues/520) Β· +[Evidence index #527](https://github.com/InferMatrixLab/InferRSI/pull/527). + +| Configuration | Speedup | Relative time reduction | +| --- | --- | --- | +| 10-step dense | 5.04Γ— | **βˆ’80.2%** | +| 10-step + Cache-DiT 0.12 | 7.37Γ— | **βˆ’86.4%** | +| 11-step dense | 4.74Γ— | **βˆ’78.9%** | +| 12-step dense | 4.34Γ— | **βˆ’77.0%** | + +TTFA is time to first audio. RTF is real-time factor. WER is word error rate. + +Upstream PR statuses checked on 2026-11-20. To refresh a row, substitute its PR +number in `gh pr view 10184 --repo vllm-project/vllm-omni --json state,isDraft`. + +## Standalone engine benchmarks + +### Standalone video/audio engines + +[h3-metal](https://github.com/zuiho-kai/h3-metal) provides Metal-native FastH3 and MiniMax-H3 text-to-video/audio inference on Apple GPUs. + +Author-reported results on **Apple M5 Max (40-core GPU), 128 GB, macOS 27**, at **768Γ—448, 49 frames, seed 0**, using real weights with one engine running at a time. + +| Engine | Device | Steps | Cold | Warm | vs. MLX (cold / warm) | vs. h3.c (cold / warm) | +| --- | --- | --- | --- | --- | --- | --- | +| `PipeNetwork/minimax-h3-mlx` (4-bit) | Apple M5 Max (40-core GPU) | 20 | 153.1 s | 134.6 s | 1.00Γ— / 1.00Γ— | 0.44Γ— / 0.54Γ— | +| `antirez/h3.c`, default preset | Apple M5 Max (40-core GPU) | 20 | 67.6 s | 73.3 s | 2.26Γ— / 1.84Γ— | 1.00Γ— / 1.00Γ— | +| **h3-metal**, original H3 | Apple M5 Max (40-core GPU) | 20 | **54.29 s** | **29.89 s** | **2.82Γ— / 4.50Γ—** | **β€” / 2.45Γ—** | +| **h3-metal**, FastH3 4-step + block-sparse attention | Apple M5 Max (40-core GPU) | 4 ΒΉ | **48.23 s** | **18.20 s** | **3.17Γ— / 7.40Γ—** | **1.40Γ— / 4.03Γ—** | + +Cold includes the first clip's weight loading, text encoding and preparation; warm reuses loaded weights and prepared state. h3-metal's cold time is 19.7% below h3.c's, just under the 20% bar this page highlights. + +ΒΉ FastH3 uses a distilled 4-step checkpoint, unlike the 20-step configurations. Warm values average clips 2–4 for the 20-step rows and two A/B/B/A warm runs for FastH3. + +[Author's report and reproduction β†’](https://github.com/zuiho-kai/h3-metal/blob/53b7fa5d88e2497ab427ab9c7155535f7e6e6817/README.md#performance). These external results were not rerun in InferRSI. + +### Standalone speech engines + +Concurrency 1 (C1) results reported by each project. Values are **engine / vLLM-Omni / SGLang-Omni**; compare within a row, since workloads and audio-frame definitions differ. + +| Project | Device | Throughput (frames/s) ↑ | TTFA p50 (ms) ↓ | Report and status | +| --- | --- | --- | --- | --- | +| [Drama 3](https://github.com/InferMatrixLab/drama-3-single-b300-engine) | 1Γ— B300 | **309.17** / 195.89 / 169.80 | **32.83** / 47.29 / 79.38 | [3-run medians](https://github.com/InferMatrixLab/drama-3-single-b300-engine/blob/58a1882a4d941f8d21bfce6be37d3d5e7c9fde80/docs/current-results.md); overall target unmet | +| [Qwen3-TTS](https://github.com/InferMatrixLab/qwen3-tts-rsi-npu-engine) | 1Γ— Ascend 910B1 | **61.12** / 12.71 / 4.31 | **50.41** / 263.51 / 439.72 | [3-run means](https://github.com/InferMatrixLab/qwen3-tts-rsi-npu-engine/blob/a29f8193b557c415175d70919bdfb9bd3dcd51fc/evidence/campaign-20261017/same-card-performance.md); source campaign accepted | +| [MiniCPM-o 4.5](https://github.com/InferMatrixLab/minicpm-o-4.5-single-gpu-engine) | 1Γ— L20X | **397.43** / 187.52 / 278.72 | **126.2** / 295.3 / 381.2 | [5-run means](https://github.com/InferMatrixLab/minicpm-o-4.5-single-gpu-engine/blob/a4f4778f312791c2d190b48e521141887ed160c8/docs/current-results.md); target unmet, quality unqualified | + +MiniCPM's SGLang baseline returns a complete waveform, so its TTFA equals end-to-end latency. These external results were not rerun in InferRSI.