From 23be5b7267ccb3d45e05d623a830fb13912b5a57 Mon Sep 17 00:00:00 2001 From: Dong1017 <55723162+Dong1017@users.noreply.github.com> Date: Fri, 20 Nov 2026 10:59:37 +0800 Subject: [PATCH 1/7] docs: refresh README quickstart and module guides Align the shared diagrams, paths and import guidance with the current package tree. Document the checkout-scoped knowledge grants and distinguish the released CLI from task entry on main. Agent-assisted-by: Codex (author) Signed-off-by: Dong1017 <55723162+Dong1017@users.noreply.github.com> --- README.md | 95 +++++++++++++------ docs/architecture.md | 38 ++++---- docs/assets/modular-components-en.svg | 95 ++++++++++--------- docs/assets/modular-directory-map-en.svg | 112 +++++++++++------------ docs/contracts/current-design.md | 4 +- docs/contributing.md | 2 +- 6 files changed, 190 insertions(+), 156 deletions(-) diff --git a/README.md b/README.md index 3eb933355..16f0500e4 100644 --- a/README.md +++ b/README.md @@ -18,9 +18,13 @@ InferRSI is a **CLI workbench for inference optimization with coding agents**. B ## πŸ“° News -Selected release milestones; see [all releases](https://github.com/InferMatrixLab/InferRSI/releases) -for packages, checksums and the complete changelog. +Milestones below highlight changes to how you run or evaluate an experiment. +The [daily release bot](docs/releases.md) publishes a preview when the default +branch's source tree changes. [All releases and checksums](https://github.com/InferMatrixLab/InferRSI/releases). +- **2026-11-20 β€” [v0.2.6](https://github.com/InferMatrixLab/InferRSI/releases/tag/v0.2.6)** added + paired optimization reports for runs with knowledge on and off, and a + service-reuse evaluation mode for project campaigns. - **2026-11-14 β€” [v0.2.0](https://github.com/InferMatrixLab/InferRSI/releases/tag/v0.2.0):** Formal release. Highlights since v0.1.1 include three packaged optimization tasks (MoE forward, Qwen-Image @@ -74,10 +78,10 @@ real-model gains remain unmeasured. [Flow, limits and reproduction β†’](docs/har | You want to… | Start at | | --- | --- | -| Choose a starting point for a new model, performance, debug, accuracy, upgrade or engine task | Choose the [primary skill for the requested outcome](docs/knowledge-layout.md#seven-deliverables) and read its procedure with `inferrsi skills enter ` before the first evidence search. Follow its decision points using the actual run and acceptance criteria. | +| Choose a procedure for a model, performance, debug, accuracy, upgrade or engine task | Read the [primary skill for your deliverable](docs/knowledge-layout.md#seven-deliverables) before searching for evidence. To open it from the CLI, use `inferrsi skills enter ` (introduced after v0.2.6). | | Modify InferRSI itself | [Contributor guide](docs/contributing.md) and [agent collaboration](docs/agent-collaboration.md), under an accepted Issue scope | | Use the Workbench to run experiments | [Quickstart](docs/quickstart.md): prepared target environment, benchmark/quality contracts, authorized experiment, recorded evidence | -| Complete an optimization benchmark in the current Agent | [Iterative task sessions](docs/optimization-benchmarks.md), with frozen rounds, independent feedback and final retesting | +| Run a packaged optimization benchmark | [Iterative task sessions](docs/optimization-benchmarks.md) default to a fresh agent conversation on the same host. Each session uses a frozen budget and independently retests its selected candidate. | | Deliver an independent inference engine | [Engine delivery template](docs/template/README.md): copyable scaffold and recipe handoff; no InferRSI runtime dependency | For every CLI/Python operation β€” discovery, normalized reads, store candidates, experiments β€” see the [operation guide](docs/operations.md). @@ -99,7 +103,7 @@ are reported directly by upstream contributors, with validation scoped to their | **Qwen-Image-2.1-Turbo** | 1Γ— Ascend 910B1 | | **βˆ’22.20%** offline generation latency
Warm repeated prompts | Upstream parent-to-merge comparison. 18/18 RGBA images were pixel-identical. The 30% target was unmet. Both arms aborted during teardown, leaving lifecycle `INCONCLUSIVE`. | [Final-merge measurements](https://github.com/vllm-project/vllm-omni/pull/8729#:~:text=Final%2Dmerge%20performance%20and%20image%20comparison) Β· [Earlier Ascend cohorts](docs/qwen-image-2.1-turbo-ascend.md) | [#8729](https://github.com/vllm-project/vllm-omni/pull/8729) (merged) | | **MiniMax-H3 Ref2VA** | 8Γ— Ascend 910B1 | | **407.27 β†’ 108.12 s (βˆ’73.45%)**
5 s video, HTTP end-to-end vs. dense inference | All 14 frozen quality comparisons passed. | [Campaign #293](https://github.com/InferMatrixLab/InferRSI/issues/293) Β· [Pinned results](evidence/h3-ref2va-ascend-293/results/final-5s/README.md) | [#9698](https://github.com/vllm-project/vllm-omni/pull/9698) (draft) | | **MiniMax-H3 Ref2VA** | 4 logical Ascend 910C devices | | **291.25 β†’ 286.67 s (βˆ’1.57%)**
5 s video, HTTP end-to-end | One lossless screening pair. Strict quality checks passed for the screened candidate. Overall campaign `INCONCLUSIVE`. | [Campaign #297](https://github.com/InferMatrixLab/InferRSI/issues/297) Β· [Evidence PR #317](https://github.com/InferMatrixLab/InferRSI/pull/317) | [#9745](https://github.com/vllm-project/vllm-omni/pull/9745) (draft) | -| **MiniMax-H3-Image** | 4Γ— Ascend 910B1 | | **16.89 β†’ 10.82 s** Β· 4736Γ—3552 generation
**4.18 β†’ 2.67 s** Β· 1024Γ—1024 generation
**5.68 β†’ 4.57 s** Β· 1024Γ—1024 editing | Single-launch HTTP medians vs. the same-layout SP4 control. All 30 numerical image screens passed. Latency targets were unmet and final qualification remains `INCONCLUSIVE`. | [Campaign #390](https://github.com/InferMatrixLab/InferRSI/issues/390) Β· [Evidence PR #394](https://github.com/InferMatrixLab/InferRSI/pull/394) | [#10184](https://github.com/vllm-project/vllm-omni/pull/10184) (open) | +| **MiniMax-H3-Image** | 4Γ— Ascend 910B1 | | **16.89 β†’ 10.82 s** Β· 4736Γ—3552 generation
**4.18 β†’ 2.67 s** Β· 1024Γ—1024 generation
**5.68 β†’ 4.57 s** Β· 1024Γ—1024 editing | Single-launch HTTP medians vs. the same-layout SP4 control. All 30 numerical image screens passed. Latency targets were unmet and final qualification remains `INCONCLUSIVE`. | [Campaign #390](https://github.com/InferMatrixLab/InferRSI/issues/390) Β· [Evidence PR #394](https://github.com/InferMatrixLab/InferRSI/pull/394) | [#10184](https://github.com/vllm-project/vllm-omni/pull/10184) (merged) | | **MiniMax-H3-Image** | 2Γ— RTX 3090 | | **36.62 β†’ 14.59 s (βˆ’60.16%)**
Warm mean HTTP delivery latency | One upstream A/B of complete deployment profiles. All nine image pairs had identical decoded RGB pixels, including warmups. Repeated qualification remains incomplete. | [Paired deployment benchmark](https://github.com/vllm-project/vllm-omni/pull/10243#:~:text=One%20paired%20deployment%20comparison) | [#10243](https://github.com/vllm-project/vllm-omni/pull/10243) (open) | | **MiniCPM-o 4.5** | 1Γ— L20X | | **+15.55%** throughput Β· **βˆ’27.69%** median TTFA
Packed conditioning, C32, same 50-frame window
**+20.5%** at C16 Β· **+25.4%** at C32
TensorRT + MPS throughput | The packed-conditioning candidate missed the exploratory WER margin against the tuning reference at the 50-frame setting. The separate TensorRT + MPS comparison had speech-quality verdict `FAIL`. | [Campaign #49](https://github.com/InferMatrixLab/InferRSI/issues/49) | [#9218](https://github.com/vllm-project/vllm-omni/pull/9218), [#9188](https://github.com/vllm-project/vllm-omni/pull/9188) (both merged) | | **Qwen3-Omni** | 2Γ— L20X | | **+7.79%** throughput Β· **βˆ’6.06%** mean RTF
One C64 pilot | Median time to first audio **regressed by 7.12%**. Campaign `INCONCLUSIVE`. | [Campaign #112](https://github.com/InferMatrixLab/InferRSI/issues/112) Β· [Pinned pilot](https://github.com/InferMatrixLab/InferRSI/tree/e1ae256fc075ea9ab64aec68da749cf4a8c22ac7/evidence/qwen3-omni-2gpu-20261023/results/pr9283-cudnn-performance) | [#9283](https://github.com/vllm-project/vllm-omni/pull/9283) (draft) | @@ -118,7 +122,8 @@ campaign and this cohort used different hardware and measurement protocols. TTFA is time to first audio. RTF is real-time factor. WER is word error rate. -Upstream PR statuses checked on 2026-11-12. +Upstream PR statuses checked on 2026-11-20. To refresh a row, substitute its PR +number in `gh pr view 10184 --repo vllm-project/vllm-omni --json state,isDraft`. @@ -164,8 +169,6 @@ MiniCPM's SGLang baseline returns a complete waveform, so its TTFA equals end-to Requires **Python 3.10+**. Licensed under [Apache-2.0](LICENSE). **InferRSI is a Developer Preview (pre-release), with no API stability promise.** -The [daily release bot](docs/releases.md) publishes a version when the default -branch changes. ### Use InferRSI in Codex @@ -218,22 +221,55 @@ then install the downloaded wheel: python -m pip install "/inferrsi--py3-none-any.whl" ``` -For CLI development from this checkout: +For a source install, run these commands from the InferRSI checkout. ```bash python -m venv .venv source .venv/bin/activate python -m pip install . +``` + +From your target engine checkout, inspect the environment and list the packaged +benchmarks. This example uses a vLLM-Omni checkout and the CPU probe. + +```bash +inferrsi --help +inferrsi doctor --repo . --repo-type vllm_omni --backend cpu +inferrsi benchmarks list +``` + +For a standalone lookup, search the packaged knowledge and follow a result's +printed `open` command to read its sources and limitations. -# Find relevant tuning knowledge and follow the printed open command. +```bash inferrsi find 'Qwen-Image-2.1 Ascend memory bottleneck' -# Export a context bundle for an agent that needs one. -inferrsi context --repo-type vllm_omni --backend cuda --output context.md ``` -For Ascend image-generation context: +`find` has one external channel, `cannbot-knowledge`, disabled by default. +From the target checkout, `inferrsi cannbot enable --checkout --python ` +authorizes local reads using that source checkout and interpreter. The source is +distributed separately, and its CANN Open Software License Agreement v2.0 governs +redistribution. MindIE has its own reader under +`inferrsi mindie`. [External channel setup](docs/external-channels.md). + +
+Local grants and revocation + +Run `inferrsi cannbot disable` or `inferrsi mindie disable` from the same +target checkout to remove its grant. The files are +`~/.config/inferrsi/cannbot-projects/.json` and +`~/.config/inferrsi/mindie-projects/.json`, respectively. +The hash identifies the target checkout path. MindIE reads its adapter +settings from `~/.config/mindie-agent/codex.json` by default. +[MindIE setup](docs/codex-plugin.md#mindie-knowledge). + +
+ +Export cited context for an agent. Both `doctor` and `context` accept +`--backend` values `cpu`, `cuda`, `rocm`, `ascend` and `mlx`. ```bash +inferrsi context --repo-type vllm_omni --backend cuda --output context.md inferrsi context --repo-type vllm_omni --backend ascend \ --scenario diffusion.image_generation --output ascend-context.md ``` @@ -244,9 +280,10 @@ Ready to measure your own model? [Connect your benchmark and quality checks β†’] ## 🧩 Architecture -**One experiment loop. Independent model, repository and hardware adapters.** -The implemented [Harness Γ— Model Γ— Infra boundary](docs/architecture.md) keeps new -campaigns out of the common AB/BA scheduler, process cleanup and evaluator. +Apps connects the CLI to host inspection, knowledge retrieval and experiment +workflows. Harness runs trials and manages process lifetimes. Packaged +benchmarks supply their workloads and assessment rules. +[Module responsibilities and dependencies](docs/architecture.md#module-responsibilities-and-dependencies). ![InferRSI system architecture: inputs, experiment loop and evidence](docs/assets/system-architecture-en.svg) @@ -260,13 +297,11 @@ not implemented. Process-cleanup behavior still needs platform-specific validati See the [source audit and proposed remedies](docs/reviews/source-audit-ac70272.md) for the 34 review points, qualifications and suggested acceptance checks. -Alongside the engine loop, the [offline improvement controller](docs/harness-improvement.md) -composes the existing Skill Eval comparator and independent verifier. Its acceptance -rules live in `benchmark/improvement.py` and its search progression in -`harness/improvement.py`; the tool adapter freezes snapshots and -carries the total budget across probes, rerun and holdout. The engine diagrams above -continue to describe the engine path; this additional lane does not provide engine -performance evidence or an autonomous model-backed optimizer. +For offline Skill Eval, the [Apps workflow](src/inferrsi/apps/workflows/improvement.py) +combines [Benchmark acceptance rules](src/inferrsi/benchmark/improvement.py) with +[Harness candidate selection](src/inferrsi/harness/improvement.py). The tool adapter +freezes snapshots and carries one budget across probes, a rerun and an independent +holdout. [Controller protocol and limits](docs/harness-improvement.md).
See the implemented modules @@ -275,6 +310,11 @@ performance evidence or an autonomous model-backed optimizer.
+Existing CLI flags, external JSON metrics, run artifacts and reports are retained. +Python callers using retired module paths must update their imports. +[Compatibility and historical evidence](docs/architecture.md#compatibility-and-historical-evidence) +Β· [Import migration list](CHANGELOG.md). +
Find the right directory to extend @@ -282,7 +322,8 @@ performance evidence or an autonomous model-backed optimizer.
-[Extension guide and compatibility boundary β†’](docs/architecture.md#where-to-extend) Β· +[Directory map](docs/architecture.md#directory-map) Β· +[Extension guide](docs/architecture.md#where-to-extend) Β· [Original design diagrams (English / Chinese) β†’](docs/architecture.md#knowledge-and-evidence-ownership) @@ -299,12 +340,10 @@ performance evidence or an autonomous model-backed optimizer. | [Qwen Ascend](docs/qwen-image-2.1-turbo-ascend.md) | NPU results, rejected rounds and verification scope | | [h3-metal lessons](docs/h3-metal-apple-silicon-lessons.md) | MiniMax-H3 on Apple silicon: measurement pitfalls, gains, rejected directions and search-loop lessons | | [Apple silicon playbook](docs/apple-silicon-playbook.md) | Measured, engine-agnostic lessons for Metal/MLX engines: unified memory, Metal execution and benchmarking a shared Mac | -| [Knowledge base](src/inferrsi/knowledge/data/skill_library/records/README.md) | Three learning tiers: contribution, dependency impact and peer techniques | -| [Skill catalog](src/inferrsi/knowledge/data/skill_library/README.md) | Searchable records linking categorized skills to full packaged procedures and resources | +| [Engineering records](src/inferrsi/knowledge/data/skill_library/records/README.md) | Source revisions, backend applicability and validation limits for contribution, dependency and peer findings | +| [Skill library](src/inferrsi/knowledge/data/skill_library/README.md) | Primary task procedures, systems, playbooks, workflows and casebooks | | [Source audit](docs/reviews/source-audit-ac70272.md) | 34 review points with proposed remedies, qualifications and validation criteria | | [Daily learning prompt](docs/daily-learning-prompt.md) | 06:00 Asia/Shanghai L2C + performance workflow and prerequisites | -| [Complete skill library](src/inferrsi/knowledge/data/skill_library/README.md) | Task entrypoint, system knowledge, playbooks, workflows and casebooks | -| [vLLM-Omni experience records](src/inferrsi/knowledge/data/skill_library/records/README.md) | Backend-scoped tuning methods, sources, and applicability | | [Contributor guide](docs/contributing.md) | Development workflow, evidence contracts, and CI | | [Engine delivery template](docs/template/README.md) | Copyable engine scaffold, branding, evidence and schema-v1 catalog recipe | diff --git a/docs/architecture.md b/docs/architecture.md index ebc5a5724..8b6db81c8 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -44,10 +44,9 @@ and [TTS service protocol](tts-benchmark.md) expose their documented execution boundaries. Apps and Harness callers assemble these operations into optimization rounds under the #362 target architecture. Each task records its qualification. -The local experiment path now has concrete Harness, Model and Infra modules. The -CLI, RunSpec, knowledge retrieval and existing command contracts remain stable. -The diagrams below describe implemented code, with the historical compatibility -boundary called out separately. +The diagrams below show where the CLI assembles an experiment, which package +owns each step and where to extend it. The [compatibility notes](#compatibility-and-historical-evidence) +cover retained interfaces and imports that moved. ## Three-objective delivery framing @@ -92,12 +91,14 @@ interfaces; existing CLI commands remain supported. ![Implemented modules and their dependencies](assets/modular-components-en.svg) -| Layer | Owns | Does not own | -| --- | --- | --- | -| Entry and knowledge | CLI arguments, immutable RunSpec, cited knowledge/context export | Model execution or automatic source editing | -| Harness | AB/BA order, deadlines, source checks, scenario composition, evidence, performance adjudication and reporting | Device-specific conditions, request schemas or kernel dispatch | -| Model | Anthropic provider adapters and their request authorization | Scheduling, worker lifetime, hardware selection, request payloads, metric contracts or quality checks | -| Infra | Repository identity, device discovery/qualification, storage and artifacts | Model constants, campaign gates or performance verdicts | +| Package | Responsibilities in this checkout | +| --- | --- | +| Apps | Parses CLI arguments, inspects the target repository and hardware, connects workflows, stores results and renders reports. | +| Knowledge | Searches packaged procedures, cases and records, and exports cited context. | +| Harness | Runs AB/BA trials, enforces deadlines, manages worker lifetimes and records execution evidence. | +| Benchmark | Supplies packaged workloads, sampling policies, metric and quality validators, and acceptance rules. | +| Model | Provides Anthropic adapters and their request authorization. | +| Infra | Provides file digests and JSON artifact writes. | Dependencies flow from entrypoints to Harness. Harness reaches no Infra or Model module, and Infra does not import Model or Harness. None @@ -114,12 +115,9 @@ repository-pack description through the neutral `resources.repository_packs` leaf, under the one permanent importer rule in [module-layout.md](module-layout.md#permanent-neutral-leaf-importer-rule-proposed). -The implemented table above is not the target four-module layout. The -suggested rule, including what the import gate does not prove, is -[module-layout.md](module-layout.md). That page does not move code. It also -records the first-wave compatibility pairs that #412 R1 retired and the -permanent neutral-leaf importer rule, both held in -[legacy-knowledge-manifest.json](legacy-knowledge-manifest.json). +The [target module layout](module-layout.md) describes the migration and the +scope of the static import checks. The [migration manifest](legacy-knowledge-manifest.json) +records the retired compatibility pairs and the permanent neutral-leaf importer rule. The caller still prepares the benchmark runtime and baseline/candidate source checkouts. Separate working directories alone do not isolate Python imports, @@ -215,6 +213,7 @@ src/inferrsi/ skill_evaluation.py Composed bundle export, trial run and summary skill_eval_engineering.py Composed artifact check: task rules around sandbox replay improvement.py Composed assessment, selection, rerun and holdout acceptance + cannbot/ Per-project grant and the external channel used by Find mindie/ project.py Opt-in per-checkout grant, the worker launch and the recorded read context.py Export supplied MindIE reads into a caller-reviewed campaign trace @@ -301,15 +300,16 @@ forwarding shell or alias: the seven Knowledge shells, whose code is in `reporting.py`, whose code is in `apps/legacy/`; and `inspection.py`, whose host binding is `apps/inspection.py`. The build-generated `_version.py` is ignored wheel metadata, not a tracked root file. -[module-layout.md](module-layout.md#retired-root-modules) maps each retired -path to its replacement and lists what still keeps the isolation gate false. +The [migration list](module-layout.md#retired-root-modules) maps each retired +path to its replacement. The [import gate](module-layout.md#import-gate) defines +the dependency checks. ## Where to extend | Add | Main extension locations | Required qualification | | --- | --- | --- | | Model or scenario | `harness/serving`, `benchmark/metrics.py`, `benchmark/validators`, `harness/scenarios` | Frozen request and metric contract; model-specific quality coverage | -| Repository | `apps/host/targets`, a composed scenario, `data/repo_packs` | Stable source identity; command/result contract; environment isolation | +| Repository | `apps/host/targets`, a composed scenario, `harness/data/repo_packs` | Stable source identity; command/result contract; environment isolation | | Hardware | `apps/host/platforms` | Device selection, process exclusivity and independent backend/model evidence; a backend is registered in `apps/host/platforms/__init__.py` and its probe command is defined in the implementing module under `apps/host/platforms/`, not in package data | The acceptance rule is: **a new model or hardware adapter must reuse the common diff --git a/docs/assets/modular-components-en.svg b/docs/assets/modular-components-en.svg index fbe0d2da0..d4a4f9d75 100644 --- a/docs/assets/modular-components-en.svg +++ b/docs/assets/modular-components-en.svg @@ -1,50 +1,47 @@ -InferRSI implemented module architectureThe entry layer composes Harness, Model and Infra. Shared modules are implemented; historical validators remain frozen. Automatic agents and distributed executors are future work. - - -INFERRSI / IMPLEMENTED ARCHITECTURE -One experiment loop. Independent extension boundaries. - -ENTRY + KNOWLEDGE -CLI β†’ immutable RunSpec β€’ Knowledge search β†’ cited manual context - - -HARNESS / ORCHESTRATION - -Runner + external scenario -AB/BA order and deadline -Source checks and trial evidence -execute / snapshot / metric hooks - -Media scenario -Warmups and seed ordering -First response + sample artifacts -Request callback; no model dispatch - -Evaluation + reporting -Performance and quality evidence -PASS / FAIL / INCONCLUSIVE -JSON records + HTML report - - -Scenario adapters compose these layers - -MODEL / CONTRACTS -Protocols + validators -Image payloads and metric interpretation -PNG/hash checks; RGB + alpha comparison -Campaign supplies model, settings and gates - -INFRA / EXECUTION -Targets + platforms + processes -Source identity; device discovery / qualification -Subprocess lifetime, HTTP timing and artifacts -No model constants or performance verdicts - -uses - -COMPATIBILITY BOUNDARY -Old CLI / import paths remain available. Historical LTX helpers and quality validators stay frozen. -New campaigns use shared modules and fresh helper hashes; old evidence keeps its original bundle. -Extend model / repository / hardware adapters without editing AB/BA, process cleanup or adjudication. -Future work: automatic planning and patching, SSH execution, distributed leases and Hub publication. + + How InferRSI assembles a benchmark + Apps connects Knowledge retrieval, Harness execution and Benchmark assessment. CLI flags and result formats survive the module moves. Python callers must update retired imports. + + + + + + + + How InferRSI assembles a benchmark + + + Apps + The CLI resolves the task and target. Workflows connect the inputs, execution and assessment. + + + + + + + + + Knowledge + Retrieves cited procedures, cases + and records when requested. + + + Harness + Executes trials, manages deadlines + and records what ran. + + + Benchmark + Supplies the packaged workload, + metrics and assessment rules. + + + Compatibility across module moves + CLI flags, external JSON metrics, run artifacts and reports are retained. Retired Python imports must be updated. diff --git a/docs/assets/modular-directory-map-en.svg b/docs/assets/modular-directory-map-en.svg index 908040f19..4d4113b8d 100644 --- a/docs/assets/modular-directory-map-en.svg +++ b/docs/assets/modular-directory-map-en.svg @@ -1,58 +1,56 @@ -InferRSI module-to-directory mapImplemented Harness, Model and Infra modules map to package directories. Model, repository and hardware additions extend adapters while shared orchestration remains unchanged. - - -INFERRSI / WHERE TO EXTEND -A directory map for the next campaign. -Package root: src/inferrsi/ β€’ All solid panels below contain implemented modules. - -harness/ -Experiment composition -runner.py AB/BA + injected ports -scenarios/external.py Command trials -Sampling -> benchmark/sampling.py -Gates -> benchmark/legacy_run_v0.py -Old run, report -> apps/legacy/ -api.py, local_process.py Execute + cleanup - -model/ -Request and quality contracts -protocols/image.py Image API + PNG response -protocols/metrics.py Metric schema checks -validators/image.py Hashes + RGB/alpha gates - -Model pins and thresholds: campaign-owned - -infra/ -Environment and resource ownership -targets/git.py Source identity -platforms/ Probes + qualification -executors/local.py Subprocess + deadline -processes.py Owned worker cleanup -http.py, artifacts.py Transport + records -inspection.py, environment.py, commands.py -campaign.py Utility import facade - -Entry + data/ -Existing contracts and reusable knowledge -cli.py, arguments.py CLI entry -runspec.py Immutable RunSpec -catalog.py Search + cited context -skill_library/records/ Findings -skill_library/config/ Discovery config - -examples/ Campaign composition -tests/fixtures/legacy/ Frozen compatibility tests - -ADD A MODEL -ADD A REPOSITORY -ADD HARDWARE -model/ + harness/scenarios/ -Freeze protocol and quality checks -infra/targets/ + repo_packs/ -Compose the benchmark scenario -infra/platforms/ (probe commands) -Qualify the backend independently - -PRESERVED COMPATIBILITY -Root execution / inspection / evaluation / reporting wrappers; historical validators and their LTX helper. + + Selected directories under src/inferrsi/ + Apps owns the CLI, host bindings and workflows. Harness runs trials, Benchmark defines and assesses tasks, and Knowledge owns retrieval and the skill library. RunSpec and the repository pack reader have separate paths. + + + + + Where to extend InferRSI + Selected paths under src/inferrsi/ + + + apps/ + cli/ + Command entry and arguments + host/ + Repository and device probes + workflows/ + Application assembly + + + harness/ + runner.py + AB/BA trials + local_process.py + Deadlines and cleanup + records/ + Campaign records and traces + + + benchmark/ + tasks/ + Packaged workloads + metrics.py + Metric contract + validators/ + Output quality checks + + + knowledge/ + catalog.py + Search and cited context + data/skill_library/ + Procedures and findings + + contracts/runspec.py + Immutable experiment specification + resources/repository_packs.py + Shared reader for repository descriptions diff --git a/docs/contracts/current-design.md b/docs/contracts/current-design.md index 50b30e585..b99dd9b38 100644 --- a/docs/contracts/current-design.md +++ b/docs/contracts/current-design.md @@ -25,14 +25,14 @@ Historical candidate text, review dispositions, and the earlier Proposed drafts ## D001 β€” One distribution, three cores plus Apps, document contracts rather than shared business DTOs -Decision: keep Knowledge, Harness, and Benchmark as sibling independent cores; Apps is the only assembler. Each core owns its types. Cross-boundary traffic uses versioned JSON envelopes with explicit conversion; there is no shared business `contracts` package. The technical-only `inferrsi.contracts` package re-exports only `RunSpec`, defined once in `contracts/runspec.py`; `contracts/claim.py` owns the same Verdict enum re-exported by Harness. These are not shared business DTOs. The old root `contracts.py`, `runspec.py`, `claim.py`, and `_json_v1.py` files are retired without forwarding shells, and so are the old root `_repo_pack_data.py` and `evidence_paths.py`. R1 (#412) retired eleven more root modules the same way: the seven Knowledge compatibility shells, the three old-run forwarders `evaluation.py`, `execution.py` and `reporting.py`, and root `inspection.py`, whose host binding is now `inferrsi.apps.inspection` over the unchanged `inferrsi.infra.inspection` facts API. [`CHANGELOG.md`](../../CHANGELOG.md) maps every retired path to its replacement and states the import and pickle impact. +Decision: keep Knowledge, Harness, and Benchmark as sibling independent cores; Apps is the only assembler. Each core owns its types. Cross-boundary traffic uses versioned JSON envelopes with explicit conversion; there is no shared business `contracts` package. The technical-only `inferrsi.contracts` package re-exports only `RunSpec`, defined once in `contracts/runspec.py`; `contracts/claim.py` owns the same Verdict enum re-exported by Harness. These are not shared business DTOs. The old root `contracts.py`, `runspec.py`, `claim.py`, and `_json_v1.py` files are retired without forwarding shells, and so are the old root `_repo_pack_data.py` and `evidence_paths.py`. R1 (#412) retired eleven more root modules the same way: the seven Knowledge compatibility shells, the three old-run forwarders `evaluation.py`, `execution.py` and `reporting.py`, and root `inspection.py`, whose host binding is now `inferrsi.apps.inspection` over the facts API in `inferrsi.apps.host.inspection`. [`CHANGELOG.md`](../../CHANGELOG.md) maps every retired path to its replacement and states the import and pickle impact. Permitted shared technical leaves are only: - `inferrsi.contracts.json_v1` β€” the canonical encoder (D002); - `inferrsi.contracts.runspec` and `inferrsi.contracts.claim` β€” the existing RunSpec and Verdict technical types, with only RunSpec re-exported by the package initializer; - `inferrsi.contracts.evidence` β€” the existing evidence segment names and pure path rules; it reads and writes no filesystem state and is not re-exported by the package initializer; -- `inferrsi.resources.repository_packs` β€” the neutral raw pack-data reader and the existing static data enumeration (D020), formerly root `inferrsi._repo_pack_data`. Its parent initializer `resources/__init__.py` imports nothing. Outside Knowledge it has one permanent importer, `inferrsi.infra.inspection`, which binds only `repository_pack`; that exact rule is not a compatibility pair. A1 (#408) removed the separate temporary entry for root `arguments.py`: the moved parser `inferrsi.apps.cli.arguments` reads the same `REPO_TYPES` object through `inferrsi.knowledge.catalog` and does not import the leaf. +- `inferrsi.resources.repository_packs` β€” the neutral raw pack-data reader and the existing static data enumeration (D020), formerly root `inferrsi._repo_pack_data`. Its parent initializer `resources/__init__.py` imports nothing. Outside Knowledge it has one permanent importer, `inferrsi.apps.host.inspection`, which binds only `repository_pack`; that exact rule is not a compatibility pair. A1 (#408) removed the separate temporary entry for root `arguments.py`: the moved parser `inferrsi.apps.cli.arguments` reads the same `REPO_TYPES` object through `inferrsi.knowledge.catalog` and does not import the leaf. The neutral leaf carries no target selection, task, scoring, scheduling, or public business DTO, and imports none of K/H/B/Apps. It still finds the Harness-owned `pack.json` through `files("inferrsi").joinpath("harness", "data", "repo_packs", ...)`, so reading it never imports Harness. Creating the evidence directories is host I/O and stays in Apps, in `inferrsi.apps.storage.evidence_layout`; that module is not a shared leaf. The list is exact: a `contracts.` or `resources.` prefix admits no other module. diff --git a/docs/contributing.md b/docs/contributing.md index 636fbde40..59cf5c770 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -36,7 +36,7 @@ Nightly checks cover Linux and Windows checkouts with `core.autocrlf` set to `false`, `true` and `input`. Use `harness/scenarios`, `harness/serving` and `benchmark/validators` for new models, -`apps/host/targets` with `data/repo_packs` for repositories, and +`apps/host/targets` with `harness/data/repo_packs` for repositories, and `apps/host/platforms` for hardware. A backend is registered in `apps/host/platforms/__init__.py`; its probe command is defined in the implementing module under `apps/host/platforms/`. That From e520d79f385d266ee13e04283b0e9dadf725f6e4 Mon Sep 17 00:00:00 2001 From: Dong1017 Date: Fri, 20 Nov 2026 18:44:47 +0800 Subject: [PATCH 2/7] docs: put a runnable quickstart and real evidence view on the first screen --- README.md | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 16f0500e4..9e87ad593 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,20 @@ -InferRSI is a **CLI workbench for inference optimization with coding agents**. Bring your benchmark and quality checks; compare implementations, trace every gain to its source, and carry useful tuning lessons into the next experiment. +InferRSI is a **CLI workbench for engineers who optimize inference**. Point it at your target engine, benchmark and quality checks, and it runs repeated A/B comparisons, gives every gain a sourced verdict, and keeps the proof in a report you can hand to a reviewer β€” instead of an untraceable screenshot. + +**Try it now.** From a fresh shell, this clones the repo, installs it into a virtual environment, and lists the packaged tasks: + +```bash +git clone https://github.com/InferMatrixLab/InferRSI.git && cd InferRSI +python -m venv .venv && source .venv/bin/activate && pip install . +inferrsi benchmarks list +# β†’ ["moe-forward-npu", "qwen21-denoise-npu", "qwen3-tts-npu"] +``` + +Prefer a coding agent? Install the [Codex plugin](docs/codex-plugin.md#install-from-the-release-zip) from [Releases](https://github.com/InferMatrixLab/InferRSI/releases) and ask a question in your repo. Every recorded run also produces a local evidence dashboard: + +![Run-review dashboard: for each round, the knowledge retrieved, the evidence selected or left out, and what was used](docs/assets/dashboard-knowledge-default-run-review.png) From e55eacfb0ed764626223afb5207377d5ecc809ed Mon Sep 17 00:00:00 2001 From: Dong1017 <55723162+Dong1017@users.noreply.github.com> Date: Fri, 20 Nov 2026 20:32:45 +0800 Subject: [PATCH 3/7] docs: clarify onboarding and curate measured gains Use aligned result tables with explicit baselines, preserve validation limits, and link the benchmark and historical records. Refresh the README banner and collaboration guidance. Signed-off-by: Dong1017 <55723162+Dong1017@users.noreply.github.com> Agent-assisted-by: Codex (author) --- README.md | 424 +++++++++----------------------- docs/assets/inferrsi-banner.svg | 98 +++++--- docs/optimization-results.md | 128 ++++++++++ 3 files changed, 307 insertions(+), 343 deletions(-) create mode 100644 docs/optimization-results.md diff --git a/README.md b/README.md index 9e87ad593..b57b7f13b 100644 --- a/README.md +++ b/README.md @@ -1,363 +1,167 @@
-![InferRSI β€” Faster inference. Verifiable progress.](docs/assets/inferrsi-banner.svg) +![InferRSI β€” an agent app to onboard and optimize models. The outer loop improves the engine and the inner loop improves the optimizer.](docs/assets/inferrsi-banner.svg) -**Turn inference bottlenecks into verified improvements.** - -[![Python 3.10+](https://img.shields.io/badge/Python-3.10%2B-3776AB?style=flat-square)](pyproject.toml) -[![CLI](https://img.shields.io/badge/Interface-CLI-14B8A6?style=flat-square)](docs/quickstart.md) -[![vLLM-Omni contributions](https://img.shields.io/badge/vLLM--Omni-Contributions-8B5CF6?style=flat-square)](#results) - -[News](#news) Β· [Why InferRSI](#why-inferrsi) Β· [How It Works](#how-it-works) Β· [Results](#results) Β· [Quickstart](#quickstart) Β· [Architecture](#architecture) Β· [Docs](#docs) +[Quickstart](#quickstart) Β· [How It Works](#how-it-works) Β· [Selected Results](#selected-results) Β· [Docs](#documentation) Β· [Contributing](#contributing)
-InferRSI is a **CLI workbench for engineers who optimize inference**. Point it at your target engine, benchmark and quality checks, and it runs repeated A/B comparisons, gives every gain a sourced verdict, and keeps the proof in a report you can hand to a reviewer β€” instead of an untraceable screenshot. +InferRSI helps coding agents and engineers profile inference workloads, test +optimizations against a baseline, and verify output quality before adopting a +change. Bring an engine checkout, a workload, and your quality checks. + +## Quickstart -**Try it now.** From a fresh shell, this clones the repo, installs it into a virtual environment, and lists the packaged tasks: +**Use the CLI.** Install from source and list the packaged benchmarks on a +development machine with Python 3.10+. ```bash -git clone https://github.com/InferMatrixLab/InferRSI.git && cd InferRSI -python -m venv .venv && source .venv/bin/activate && pip install . +git clone https://github.com/InferMatrixLab/InferRSI.git +cd InferRSI +python -m venv .venv +source .venv/bin/activate +python -m pip install . inferrsi benchmarks list -# β†’ ["moe-forward-npu", "qwen21-denoise-npu", "qwen3-tts-npu"] ``` -Prefer a coding agent? Install the [Codex plugin](docs/codex-plugin.md#install-from-the-release-zip) from [Releases](https://github.com/InferMatrixLab/InferRSI/releases) and ask a question in your repo. Every recorded run also produces a local evidence dashboard: - -![Run-review dashboard: for each round, the knowledge retrieved, the evidence selected or left out, and what was used](docs/assets/dashboard-knowledge-default-run-review.png) - - - -## πŸ“° News - -Milestones below highlight changes to how you run or evaluate an experiment. -The [daily release bot](docs/releases.md) publishes a preview when the default -branch's source tree changes. [All releases and checksums](https://github.com/InferMatrixLab/InferRSI/releases). +From a target vLLM-Omni checkout, run the CPU readiness probe. -- **2026-11-20 β€” [v0.2.6](https://github.com/InferMatrixLab/InferRSI/releases/tag/v0.2.6)** added - paired optimization reports for runs with knowledge on and off, and a - service-reuse evaluation mode for project campaigns. -- **2026-11-14 β€” [v0.2.0](https://github.com/InferMatrixLab/InferRSI/releases/tag/v0.2.0):** - Formal release. Highlights since - v0.1.1 include three packaged optimization tasks (MoE forward, Qwen-Image - denoising and Qwen3-TTS), a unified optimization lifecycle, richer trajectory - and evidence dashboards, and expanded multimodal case studies. - [Release highlights](docs/release-notes/v0.2.0.md). Case studies retain their - workload-specific validation limits. -- **2026-11-01 β€” [v0.1.0 Developer Preview](https://github.com/InferMatrixLab/InferRSI/releases/tag/v0.1.0):** - First packaged preview release of the inference-optimization workbench. - - - -## ⚑ Why InferRSI - -- **Profile first.** Ground each optimization in measured bottlenecks and sourced tuning knowledge. -- **Prove the gain.** Run repeated A/B comparisons with quality gates, source fingerprints, and an HTML evidence report. -- **Keep the lessons.** Search prior experience and export cited context for Codex or other coding agents. - -You drive profiling and code changes; InferRSI runs and evaluates the benchmark and validation commands you provide. +```bash +inferrsi doctor --repo . --repo-type vllm_omni --backend cpu +``` - +Setup and task discovery run without starting a benchmark. For a target host, +select its backend (`cpu`, `cuda`, `rocm`, `ascend`, or `mlx`). +[Install guide](docs/quickstart.md#install-and-smoke) Β· [Release wheels and checksums](https://github.com/InferMatrixLab/InferRSI/releases) -## πŸ” How It Works +**Use a coding agent.** Install the plugin and ask it to check the target +checkout, workload, and quality criteria before starting an experiment. +[Codex](docs/codex-plugin.md#install-from-the-release-zip) Β· [DeepSeek Harness](docs/dsh-plugin.md) Β· [WorkSwarm](docs/workswarm-plugin.md#download-and-install) -**Optimize with evidence. Reuse reviewed lessons.** Humans and coding agents -prepare candidates; Workbench runs controlled comparisons against the target -engine. Evidence guides the next experiment and may inform reusable knowledge, -including limitations from failed attempts. -[Architecture](#architecture) Β· [Roadmap](https://github.com/InferMatrixLab/InferRSI/issues/61). +**Run your first experiment.** [Try a packaged benchmark](docs/optimization-benchmarks.md#start-the-complete-workflow) Β· [Bring your own workload](docs/quickstart.md#real-remote-experiment) - - - - InferRSI experiment and knowledge loops: humans or coding agents declare a task; Workbench compares an external target engine; evidence supports another experiment and optional reviewed lessons; candidate storage requires separate authorization - - + -Illustrative target workflow; no measured results. [Watch the MP4](docs/assets/inferrsi-loop.mp4). +## How It Works -Knowledge retrieval is read-only; candidate storage is optional and separately -authorized. Automatic model-backed search and automatic knowledge promotion remain -planned. Missing or incomplete evidence stays `INCONCLUSIVE`. +**Outer loop.** To improve the engine, the agent profiles the workload and +proposes a change. InferRSI checks quality and measures the candidate against +the baseline under the same frozen protocol. Round evidence guides the next +change. A fresh process retests the best candidate at the end of the campaign. -The **offline harness-improvement MVP** adds a separate Skill Eval path for supplied -declarative candidates. Its fixed-output experiments validate the controller; -real-model gains remain unmeasured. [Flow, limits and reproduction β†’](docs/harness-improvement.md) +**Inner loop.** To improve the optimizer, compare paired campaigns with +knowledge on and off to measure whether access to the library helps the agent. +Reviewed PRs add findings to the library for agents to consult in later +campaigns. +[Knowledge comparison](docs/benchmark-comparison.md) - +- **Knowledge** supplies source-linked skills, prior measurements, and implementation notes for the agent to consult. +- **Harness** freezes the experiment inputs and gates, runs the rounds, and records the verdict and evidence. +- **Benchmark** supplies packaged tasks, measurements, and acceptance rules. Project campaigns supply their own workload and checks. -## 🧭 Choose your starting path +**Apps** assembles these capabilities into the CLI, client integrations, and +result views. +[Module responsibilities](docs/architecture.md#module-responsibilities-and-dependencies) Β· [Workflow diagram](docs/optimization-workflow-unification.md#既序图) Β· [Architecture decision #362](https://github.com/InferMatrixLab/InferRSI/issues/362#issuecomment-6722325612) Β· [Apps migration #371](https://github.com/InferMatrixLab/InferRSI/issues/371) -| You want to… | Start at | -| --- | --- | -| Choose a procedure for a model, performance, debug, accuracy, upgrade or engine task | Read the [primary skill for your deliverable](docs/knowledge-layout.md#seven-deliverables) before searching for evidence. To open it from the CLI, use `inferrsi skills enter ` (introduced after v0.2.6). | -| Modify InferRSI itself | [Contributor guide](docs/contributing.md) and [agent collaboration](docs/agent-collaboration.md), under an accepted Issue scope | -| Use the Workbench to run experiments | [Quickstart](docs/quickstart.md): prepared target environment, benchmark/quality contracts, authorized experiment, recorded evidence | -| Run a packaged optimization benchmark | [Iterative task sessions](docs/optimization-benchmarks.md) default to a fresh agent conversation on the same host. Each session uses a frozen budget and independently retests its selected candidate. | -| Deliver an independent inference engine | [Engine delivery template](docs/template/README.md): copyable scaffold and recipe handoff; no InferRSI runtime dependency | +**Review a run in the dashboard.** Compare baseline and candidate profiles and +trace each experiment back to its evidence. From a recorded run directory, +generate a local report with `inferrsi dashboard --input . --output report.html`. +[Dashboard guide](docs/dashboard.md#generate-and-open) Β· [Download the interactive example](docs/examples/dashboard-profile-review.html) -For every CLI/Python operation β€” discovery, normalized reads, store candidates, experiments β€” see the [operation guide](docs/operations.md). +

+ + InferRSI dashboard showing baseline and candidate profiling captures and links to experiment and knowledge traces + +
+ Dashboard demo with synthetic data. +

-## πŸ“Š InferRSI Γ— vLLM-Omni: Optimization Results - -InferRSI campaigns and upstream experiments cover image, video and speech models. -Each result applies to its linked workload and measured revisions. Some results -are reported directly by upstream contributors, with validation scoped to their benchmarks. - -
-Benchmarks, validation limits, and upstream PRs - -| Model | Hardware | Optimization approach | Measured gains | Validation and limits | Evidence | vLLM-Omni PR | -| --- | --- | --- | --- | --- | --- | --- | -| **Qwen-Image-2.1-Turbo** | 4Γ— B300 |
  • Text-conditioning reuse
  • BF16 operator fusion
  • Symmetric-memory Ulysses with batched QKV exchange
| **384.16 β†’ 266.60 ms (βˆ’30.60%)**
Warm HTTP end-to-end latency, repeated prompts | Historical InferRSI campaign. Latencies are medians of five trial means from alternating AB/BA groups. All 44 RGBA images were pixel-identical. | [Campaign #3](https://github.com/InferMatrixLab/InferRSI/issues/3) Β· [Pinned results](docs/qwen-image-2.1-turbo-b300.md) | [#8739](https://github.com/vllm-project/vllm-omni/pull/8739) (merged) | -| **Qwen-Image-2.1-Turbo** | 1Γ— Ascend 910B1 |
  • Conditioning cache for repeated prompts
  • Shared modulation once per denoising step
  • Native interleaved NPU RoPE with frequency-table reuse
| **βˆ’22.20%** offline generation latency
Warm repeated prompts | Upstream parent-to-merge comparison. 18/18 RGBA images were pixel-identical. The 30% target was unmet. Both arms aborted during teardown, leaving lifecycle `INCONCLUSIVE`. | [Final-merge measurements](https://github.com/vllm-project/vllm-omni/pull/8729#:~:text=Final%2Dmerge%20performance%20and%20image%20comparison) Β· [Earlier Ascend cohorts](docs/qwen-image-2.1-turbo-ascend.md) | [#8729](https://github.com/vllm-project/vllm-omni/pull/8729) (merged) | -| **MiniMax-H3 Ref2VA** | 8Γ— Ascend 910B1 |
  • Lossy Cache-DiT profile tuning
| **407.27 β†’ 108.12 s (βˆ’73.45%)**
5 s video, HTTP end-to-end vs. dense inference | All 14 frozen quality comparisons passed. | [Campaign #293](https://github.com/InferMatrixLab/InferRSI/issues/293) Β· [Pinned results](evidence/h3-ref2va-ascend-293/results/final-5s/README.md) | [#9698](https://github.com/vllm-project/vllm-omni/pull/9698) (draft) | -| **MiniMax-H3 Ref2VA** | 4 logical Ascend 910C devices |
  • Ref2VA migration changes
  • Opt-in BF16 prefix attention
| **291.25 β†’ 286.67 s (βˆ’1.57%)**
5 s video, HTTP end-to-end | One lossless screening pair. Strict quality checks passed for the screened candidate. Overall campaign `INCONCLUSIVE`. | [Campaign #297](https://github.com/InferMatrixLab/InferRSI/issues/297) Β· [Evidence PR #317](https://github.com/InferMatrixLab/InferRSI/pull/317) | [#9745](https://github.com/vllm-project/vllm-omni/pull/9745) (draft) | -| **MiniMax-H3-Image** | 4Γ— Ascend 910B1 |
  • Opt-in VAE tile graphs
  • Approximate encoder/decoder tile batching
| **16.89 β†’ 10.82 s** Β· 4736Γ—3552 generation
**4.18 β†’ 2.67 s** Β· 1024Γ—1024 generation
**5.68 β†’ 4.57 s** Β· 1024Γ—1024 editing | Single-launch HTTP medians vs. the same-layout SP4 control. All 30 numerical image screens passed. Latency targets were unmet and final qualification remains `INCONCLUSIVE`. | [Campaign #390](https://github.com/InferMatrixLab/InferRSI/issues/390) Β· [Evidence PR #394](https://github.com/InferMatrixLab/InferRSI/pull/394) | [#10184](https://github.com/vllm-project/vllm-omni/pull/10184) (merged) | -| **MiniMax-H3-Image** | 2Γ— RTX 3090 |
  • Rank-local layerwise offload
  • Selected DiT weights kept resident
  • AdaLN caching
  • Request-local text refinement reuse
| **36.62 β†’ 14.59 s (βˆ’60.16%)**
Warm mean HTTP delivery latency | One upstream A/B of complete deployment profiles. All nine image pairs had identical decoded RGB pixels, including warmups. Repeated qualification remains incomplete. | [Paired deployment benchmark](https://github.com/vllm-project/vllm-omni/pull/10243#:~:text=One%20paired%20deployment%20comparison) | [#10243](https://github.com/vllm-project/vllm-omni/pull/10243) (open) | -| **MiniCPM-o 4.5** | 1Γ— L20X |
  • Packed conditioning on a tuned configuration
  • TensorRT speaker extraction + CUDA MPS (separate comparison)
| **+15.55%** throughput Β· **βˆ’27.69%** median TTFA
Packed conditioning, C32, same 50-frame window
**+20.5%** at C16 Β· **+25.4%** at C32
TensorRT + MPS throughput | The packed-conditioning candidate missed the exploratory WER margin against the tuning reference at the 50-frame setting. The separate TensorRT + MPS comparison had speech-quality verdict `FAIL`. | [Campaign #49](https://github.com/InferMatrixLab/InferRSI/issues/49) | [#9218](https://github.com/vllm-project/vllm-omni/pull/9218), [#9188](https://github.com/vllm-project/vllm-omni/pull/9188) (both merged) | -| **Qwen3-Omni** | 2Γ— L20X |
  • cuDNN codec autotuning in the MRv2 deployment
| **+7.79%** throughput Β· **βˆ’6.06%** mean RTF
One C64 pilot | Median time to first audio **regressed by 7.12%**. Campaign `INCONCLUSIVE`. | [Campaign #112](https://github.com/InferMatrixLab/InferRSI/issues/112) Β· [Pinned pilot](https://github.com/InferMatrixLab/InferRSI/tree/e1ae256fc075ea9ab64aec68da749cf4a8c22ac7/evidence/qwen3-omni-2gpu-20261023/results/pr9283-cudnn-performance) | [#9283](https://github.com/vllm-project/vllm-omni/pull/9283) (draft) | -| **LFM3-Omni-2B** | 1Γ— Ascend 910B1 |
  • Depth graphs
  • Grouped greedy sampling
| **0.285 β†’ 0.593 requests/s**
Upstream offline Omni API A/B, C4, temperature 0 | Retained audio waveforms matched bit-for-bit within each tested temperature. | [Offline A/B benchmark](https://github.com/vllm-project/vllm-omni/pull/9827#:~:text=Performance%20results) | [#9827](https://github.com/vllm-project/vllm-omni/pull/9827) (merged) | -| **LingBot-World** | 4Γ— Ascend 910C1 |
  • Whole-block paged KV gather with index_select
| **108.29 β†’ 103.89 s (βˆ’4.06%)**
Full-request latency, upstream AB/BA | Pre-codec tensors and decoded RGB frames matched exactly. | [KV gather benchmark](https://github.com/vllm-project/vllm-omni/pull/8112#:~:text=Performance%3A%20whole%2Dblock%20index_select%20KV%20gather%20on%20NPU) | [#8112](https://github.com/vllm-project/vllm-omni/pull/8112) (merged) | - -**Earlier Qwen-Image-2.1-Turbo Ascend measurements.** On one Ascend 910B1 at -1280Γ—704, BF16, eager, and eight native steps, the author reported a 3111.5 ms -baseline. Conditioning reuse with interleaved RoPE reached **2459.6 ms (βˆ’21.0%)** -with pixel-identical RGBA. Adding the opt-in VAE fastpath reached **2425.5 ms -(βˆ’22.0%)** and passed the reported quality gate with changed pixels. Each mean -covers three requests after two warmups. InferRSI verification covered the -import, so acceptance remains `INCONCLUSIVE`. The 30% target was unmet. The B300 -campaign and this cohort used different hardware and measurement protocols. -[Earlier Ascend report](docs/qwen-image-2.1-turbo-ascend.md) - -TTFA is time to first audio. RTF is real-time factor. WER is word error rate. - -Upstream PR statuses checked on 2026-11-20. To refresh a row, substitute its PR -number in `gh pr view 10184 --repo vllm-project/vllm-omni --json state,isDraft`. - -
- - +## Selected Results -## πŸ§ͺ Standalone engine benchmarks +Model bring-up and measured optimizations across text, image, video, and speech. +Performance tables highlight metrics that improved by at least 20% against the +listed baseline. The evidence column gives the scope and validation status. -### Standalone video/audio engines +### Model onboarding -[h3-metal](https://github.com/zuiho-kai/h3-metal) provides Metal-native FastH3 and MiniMax-H3 text-to-video/audio inference on Apple GPUs. +| Model | Engine and hardware | Milestone | Evidence | +| --- | --- | --- | --- | +| **DeepSeek V4.1 Flash** | vLLM-Ascend Β· 8Γ— Ascend 910C | W8A8 service ran the 8K-input / 1K-output workload. Six independent service runs established a baseline of **39.28 / 44.07 / 56.86 ms/token** at concurrency 1 / 4 / 16. The 25 ms/token target remains unmet. | [Bring-up and baseline](https://github.com/InferMatrixLab/InferRSI/issues/353#:~:text=9be3c22ab5dade39c0d33d49540ae021ff74bbd1) | +| **Qwen3.8-27B** | Standalone nano-vLLM-derived engine Β· 1Γ— H20 | The FP8 text service passed scoped HTTP/SSE lifecycle checks. Model-quality and performance qualification remained incomplete. | [Service bring-up](https://github.com/InferMatrixLab/InferRSI/issues/57) Β· [Performance campaign](https://github.com/InferMatrixLab/InferRSI/issues/67) | +| **LFM3-Omni-2B** | vLLM-Omni Β· 1Γ— Ascend 910B1 | Two real-weight serving tests passed. A functional eager adapter provided a baseline for the offline audio benchmark. | [Serving checks and A/B benchmark](https://github.com/vllm-project/vllm-omni/pull/9827) | -Author-reported results on **Apple M5 Max (40-core GPU), 128 GB, macOS 27**, at **768Γ—448, 49 frames, seed 0**, using real weights with one engine running at a time. +### Text serving -| Engine | Device | Steps | Cold | Warm | vs. MLX (cold / warm) | vs. h3.c (cold / warm) | -| --- | --- | --- | --- | --- | --- | --- | -| `PipeNetwork/minimax-h3-mlx` (4-bit) | Apple M5 Max (40-core GPU) | 20 | 153.1 s | 134.6 s | 1.00Γ— / 1.00Γ— | 0.44Γ— / 0.54Γ— | -| `antirez/h3.c`, default preset | Apple M5 Max (40-core GPU) | 20 | 67.6 s | 73.3 s | 2.26Γ— / 1.84Γ— | 1.00Γ— / 1.00Γ— | -| **h3-metal**, original H3 | Apple M5 Max (40-core GPU) | 20 | **54.29 s** | **29.89 s** | **2.82Γ— / 4.50Γ—** | **1.25Γ— / 2.45Γ—** | -| **h3-metal**, FastH3 4-step + block-sparse attention | Apple M5 Max (40-core GPU) | 4 ΒΉ | **48.23 s** | **18.20 s** | **3.17Γ— / 7.40Γ—** | **1.40Γ— / 4.03Γ—** | - -Cold includes the first clip's weight loading, text encoding and preparation; warm reuses loaded weights and prepared state. - -ΒΉ FastH3 uses a distilled 4-step checkpoint, unlike the 20-step configurations. Warm values average clips 2–4 for the 20-step rows and two A/B/B/A warm runs for FastH3. - -[Author's report and reproduction β†’](https://github.com/zuiho-kai/h3-metal/blob/53b7fa5d88e2497ab427ab9c7155535f7e6e6817/README.md#performance). These external results were not rerun in InferRSI. - -### Standalone speech engines - -Concurrency 1 (C1) results reported by each project. Values are **engine / vLLM-Omni / SGLang-Omni**; compare within a row, since workloads and audio-frame definitions differ. - -| Project | Device | Throughput (frames/s) ↑ | TTFA p50 (ms) ↓ | Report and status | +| Model | Measured change | Metric and hardware | Baseline | Validation and evidence | | --- | --- | --- | --- | --- | -| [Drama 3](https://github.com/InferMatrixLab/drama-3-single-b300-engine) | 1Γ— B300 | **309.17** / 195.89 / 169.80 | **32.83** / 47.29 / 79.38 | [3-run medians](https://github.com/InferMatrixLab/drama-3-single-b300-engine/blob/58a1882a4d941f8d21bfce6be37d3d5e7c9fde80/docs/current-results.md); overall target unmet | -| [Qwen3-TTS](https://github.com/InferMatrixLab/qwen3-tts-rsi-npu-engine) | 1Γ— Ascend 910B1 | **61.12** / 12.71 / 4.31 | **50.41** / 263.51 / 439.72 | [3-run means](https://github.com/InferMatrixLab/qwen3-tts-rsi-npu-engine/blob/a29f8193b557c415175d70919bdfb9bd3dcd51fc/evidence/campaign-20261017/same-card-performance.md); source campaign accepted | -| [MiniCPM-o 4.5](https://github.com/InferMatrixLab/minicpm-o-4.5-single-gpu-engine) | 1Γ— L20X | **397.43** / 187.52 / 278.72 | **126.2** / 295.3 / 381.2 | [5-run means](https://github.com/InferMatrixLab/minicpm-o-4.5-single-gpu-engine/blob/a4f4778f312791c2d190b48e521141887ed160c8/docs/current-results.md); target unmet, quality unqualified | - -MiniCPM's SGLang baseline returns a complete waveform, so its TTFA equals end-to-end latency. These external results were not rerun in InferRSI. - - - -## πŸš€ Quickstart - -Requires **Python 3.10+**. Licensed under [Apache-2.0](LICENSE). - -**InferRSI is a Developer Preview (pre-release), with no API stability promise.** - -### Use InferRSI in Codex - -Install the plugin in two steps. Use a local Codex client with ZIP upload support, -Python 3.10+ and Git. - -1. **Download the package from GitHub.** Open - [Releases](https://github.com/InferMatrixLab/InferRSI/releases) and download - `inferrsi-codex-plugin-vX.Y.Z.zip` from the selected version's assets. -2. **Install it in Codex.** If your client offers **Plugins β†’ Add β†’ Upload plugin - archive**, use it to import the downloaded ZIP. Select **Install plugin** on - the InferRSI detail page if that action is available. Once installed, enable - InferRSI and start a new conversation in your target repository. - -[Installation video (English)](docs/assets/inferrsi-codex-install-en.mp4) - -The runtime and knowledge are bundled; core operations need no separate pip -install or InferRSI source checkout. The [plugin guide](docs/codex-plugin.md#install-from-the-release-zip) -covers checksum verification and client compatibility. The video highlights the -click order with English UI illustrations based on the supplied Codex screenshots; -it does not record a completed installation. - -Try a first task: - -> Use InferRSI to research this engine's memory bottleneck. Find relevant -> knowledge and explain what is needed for a controlled comparison. - -Or inspect existing evidence: +| **DeepSeek V4.1 Flash** | **+54.45% / +36.12% / +20.80%** | Per-card throughput for prefill / decode / end-to-end decode output Β· Ascend A3, W4A8, 1P1D | 68,239 / 1,711.98 / 1,132.9 tokens/s per card | InferRSI recheck at each scenario's latency limit. The decode result was rechecked once. [Workloads and evidence](https://github.com/jiaran-king/dsv4.1-optimize) | -> Use InferRSI to inspect the existing run I provide and explain its verdict, -> missing evidence and next action. +### Image and video generation -Setup checks readiness without starting benchmarks. Experiments run locally on -the prepared target machine using your accepted gates and budget. See the -[plugin guide](docs/codex-plugin.md#install-from-the-release-zip) for installation, -updates and troubleshooting. - -### Use InferRSI in other clients +| Model | Measured change | Metric and hardware | Baseline | Validation and evidence | +| --- | --- | --- | --- | --- | +| **MiniMax-H3 Ref2VA** | **βˆ’73.45%** | 5 s video, HTTP latency Β· 8Γ— Ascend 910B1 | Dense inference Β· 407.27 s | **3.77Γ—** with lossy Cache-DiT. All 14 quality comparisons passed. [Campaign](https://github.com/InferMatrixLab/InferRSI/issues/293) Β· [Benchmark](evidence/h3-ref2va-ascend-293/results/final-5s/README.md) | +| **MiniMax-H3 Ref2VA** | **βˆ’76.3%** | 5 s video, 50-step denoising time Β· 4Γ— Ascend 950PR | In-run dense control Β· 701.95 s | **4.22Γ—** in five retained A/B slots. Historical record with inconclusive performance and quality qualification. [Archive record](https://github.com/InferMatrixLab/InferRSI/issues/520) Β· [Recipe benchmark](https://github.com/vllm-project/vllm-omni/pull/9955#:~:text=in%2Drun%20dense%20median) | +| **MiniMax-H3-Image** | **βˆ’60.16%** | Warm HTTP latency Β· 2Γ— RTX 3090 | Original layer offload Β· 36.62 s | One paired comparison with 9 pixel-identical outputs. [Upstream results](https://github.com/vllm-project/vllm-omni/pull/10243#:~:text=One%20paired%20deployment%20comparison) | +| **MiniMax-H3-Image** | **βˆ’35.9% / βˆ’36.1%** | HTTP generation latency at 4736Γ—3552 / 1024Γ—1024 Β· 4Γ— Ascend 910B1 | Same-layout SP4 control Β· 16.89 / 4.18 s | All 30 numerical checks passed across the tested workloads. Latency targets were unmet. [Benchmark](https://github.com/InferMatrixLab/InferRSI/pull/394) | +| **Qwen-Image-2.1-Turbo** | **βˆ’30.60%** | Warm HTTP latency Β· 4Γ— B300 | Pinned control Β· 384.16 ms | 44 pixel-identical outputs. [InferRSI results](docs/qwen-image-2.1-turbo-b300.md) | +| **Qwen-Image-2.1-Turbo** | **βˆ’22.20%** | Warm offline generation latency Β· 1Γ— Ascend 910B1 | Upstream parent Β· 2,921.23 ms | Upstream reference with 18 pixel-identical outputs. Teardown validation was inconclusive. [Upstream results](https://github.com/vllm-project/vllm-omni/pull/8729#:~:text=Final%2Dmerge%20performance%20and%20image%20comparison) | -For DeepSeek Harness, build and install the [native DSH bundle](docs/dsh-plugin.md). -For openJiuwen WorkSwarm v0.2.7, use the [native plugin ZIP](docs/workswarm-plugin.md#download-and-install). +### Speech and multimodal serving -### Use the CLI +| Model | Measured change | Metric and hardware | Baseline | Validation and evidence | +| --- | --- | --- | --- | --- | +| **LFM3-Omni-2B** | **+108.4%** | Offline greedy request throughput Β· 1Γ— Ascend 910B1, C4 | Functional eager adapter Β· 0.285 req/s | Retained audio matched bit-for-bit within each tested temperature. [Upstream results](https://github.com/vllm-project/vllm-omni/pull/9827#:~:text=Performance%20results) | +| **MiniCPM-o 4.5** | **+100.6% / βˆ’50.72%** | Request throughput / mean real-time factor Β· 1Γ— L20X, C32 | Pinned upstream main Β· 11.46 req/s / 0.65548 RTF | Five matched runs per arm. Formal speech-quality non-inferiority remains unestablished. [Campaign](https://github.com/InferMatrixLab/InferRSI/issues/431) Β· [Benchmark](https://github.com/vllm-project/vllm-omni/pull/10331) | +| **MiniCPM-o 4.5** | **βˆ’27.69%** | Median time to first audio Β· 1Γ— L20X, C32 | Tuned 50-frame deployment before packed conditioning | Packed conditioning missed the exploratory word error rate margin. [Benchmark](https://github.com/vllm-project/vllm-omni/pull/9218) | +| **MiniCPM-o 4.5** | **+20.5% / +25.4%** | Throughput at C16 / C32 Β· 1Γ— L20X | Pinned upstream main before TensorRT + MPS | The speech quality check failed. [Benchmark](https://github.com/vllm-project/vllm-omni/pull/9188) | +| **Qwen3-TTS 1.7B** | **+158%** | Streaming audio throughput Β· 1Γ— Ascend 910B1, C1 | Fresh no-PR control Β· 1.869 audio s/s | Historical upstream reference. Mandarin speaker similarity and some ASR scores regressed. [Upstream benchmark](https://github.com/vllm-project/vllm-omni/pull/9068) | +| **Drama 3** | **+31% / βˆ’23%** | Throughput / mean real-time factor Β· 2Γ— Ascend 910B1, C1 | Piecewise graphs Β· 17.31 frames/s / 1.237 RTF | Two runs per arm. The real-time factor targets were unmet. [Campaign](https://github.com/InferMatrixLab/InferRSI/issues/132) Β· [Benchmark](https://github.com/vllm-project/vllm-omni/pull/9308) | -This preview is not on PyPI. Download the wheel and `SHA256SUMS` from the selected -[Release](https://github.com/InferMatrixLab/InferRSI/releases), verify its checksum, -then install the downloaded wheel: + -```bash -python -m pip install "/inferrsi--py3-none-any.whl" -``` +### Standalone engines -For a source install, run these commands from the InferRSI checkout. +Engine projects report these comparisons. Speech benchmarks use one concurrent +request per engine and measure audio frames per second and median time to first audio. -```bash -python -m venv .venv -source .venv/bin/activate -python -m pip install . -``` +| Engine | Measured change | Metric and hardware | Baseline | Benchmark scope and evidence | +| --- | --- | --- | --- | --- | +| **Qwen3-TTS** | **+381% / βˆ’81%** | Audio throughput / time to first audio Β· 1Γ— Ascend 910B1 | vLLM-Omni Β· 12.71 frames/s / 263.51 ms | Three independent runs per engine. Candidate audio and contract checks passed. [Same-card benchmark](https://github.com/InferMatrixLab/qwen3-tts-rsi-npu-engine/blob/a29f8193b557c415175d70919bdfb9bd3dcd51fc/evidence/campaign-20261017/same-card-performance.md) | +| **MiniCPM-o 4.5** | **+112% / βˆ’57%** | Audio throughput / time to first audio Β· 1Γ— L20X | vLLM-Omni Β· 187.52 frames/s / 295.3 ms | Performance target unmet and quality unqualified. [Same-card benchmark](https://github.com/InferMatrixLab/minicpm-o-4.5-single-gpu-engine/blob/a4f4778f312791c2d190b48e521141887ed160c8/docs/current-results.md) | +| **Drama 3** | **+58% / βˆ’31%** | Audio throughput / time to first audio Β· 1Γ— B300 | vLLM-Omni Β· 195.89 frames/s / 47.29 ms | Three-run medians. The overall target was unmet. [Same-card benchmark](https://github.com/InferMatrixLab/drama-3-single-b300-engine/blob/58a1882a4d941f8d21bfce6be37d3d5e7c9fde80/docs/current-results.md) | +| **h3-metal** | **βˆ’59%** | Warm video generation latency Β· Apple M5 Max | h3.c Β· 73.3 s | Both engines used the original MiniMax-H3 at 20 steps. [Author benchmark](https://github.com/zuiho-kai/h3-metal/blob/53b7fa5d88e2497ab427ab9c7155535f7e6e6817/README.md#performance) | -From your target engine checkout, inspect the environment and list the packaged -benchmarks. This example uses a vLLM-Omni checkout and the CPU probe. +[Run your first experiment](#quickstart) Β· [Benchmark details and historical records](docs/optimization-results.md) Β· [vLLM-Omni model optimization roadmap](https://github.com/InferMatrixLab/InferRSI/issues/416) -```bash -inferrsi --help -inferrsi doctor --repo . --repo-type vllm_omni --backend cpu -inferrsi benchmarks list -``` + -For a standalone lookup, search the packaged knowledge and follow a result's -printed `open` command to read its sources and limitations. +## Documentation -```bash -inferrsi find 'Qwen-Image-2.1 Ascend memory bottleneck' -``` +- **Run and review.** [Operations](docs/operations.md) Β· [Packaged optimization benchmarks](docs/optimization-benchmarks.md) Β· [Result delivery](docs/result-delivery.md) +- **Use knowledge.** [Skill library](src/inferrsi/knowledge/data/skill_library/README.md) Β· [CANNbot setup and revocation](docs/external-channels.md#setting-one-up) Β· [MindIE setup and revocation](docs/codex-plugin.md#enable-and-revoke) +- **Understand and extend.** [Module layout and import migration](docs/module-layout.md) Β· [Architecture and directory map](docs/architecture.md) Β· [Standalone engine template](docs/template/README.md) +- **Follow changes.** [Releases](https://github.com/InferMatrixLab/InferRSI/releases) Β· [Changelog](CHANGELOG.md) -`find` has one external channel, `cannbot-knowledge`, disabled by default. -From the target checkout, `inferrsi cannbot enable --checkout --python ` -authorizes local reads using that source checkout and interpreter. The source is -distributed separately, and its CANN Open Software License Agreement v2.0 governs -redistribution. MindIE has its own reader under -`inferrsi mindie`. [External channel setup](docs/external-channels.md). +## Contributing -
-Local grants and revocation +**People decide in issues; agents write the PRs.** -Run `inferrsi cannbot disable` or `inferrsi mindie disable` from the same -target checkout to remove its grant. The files are -`~/.config/inferrsi/cannbot-projects/.json` and -`~/.config/inferrsi/mindie-projects/.json`, respectively. -The hash identifies the target checkout path. MindIE reads its adapter -settings from `~/.config/mindie-agent/codex.json` by default. -[MindIE setup](docs/codex-plugin.md#mindie-knowledge). +People agree on the scope, resources, and acceptance criteria in the issue. +Agents implement the agreed scope, run tests, and submit PRs with evidence. +To merge, a PR must pass the required checks and receive two independent +approvals. The issue owner uses the evidence to accept the work or reopen it. -
+Start with a reproducible workload, a baseline, and evidence of the bottleneck. +Sign off commits with `git commit -s`. +[Open an issue](https://github.com/InferMatrixLab/InferRSI/issues/new/choose) Β· [Human–agent collaboration](docs/agent-collaboration.md) Β· [Contributor guide](docs/contributing.md) Β· [License](LICENSE) -Export cited context for an agent. Both `doctor` and `context` accept -`--backend` values `cpu`, `cuda`, `rocm`, `ascend` and `mlx`. +Run the local development checks before submitting a change. ```bash -inferrsi context --repo-type vllm_omni --backend cuda --output context.md -inferrsi context --repo-type vllm_omni --backend ascend \ - --scenario diffusion.image_generation --output ascend-context.md +python -m pip install -e ".[dev]" +pytest +ruff check src tests ``` - -Ready to measure your own model? [Connect your benchmark and quality checks β†’](docs/quickstart.md#real-remote-experiment) - - - -## 🧩 Architecture - -Apps connects the CLI to host inspection, knowledge retrieval and experiment -workflows. Harness runs trials and manages process lifetimes. Packaged -benchmarks supply their workloads and assessment rules. -[Module responsibilities and dependencies](docs/architecture.md#module-responsibilities-and-dependencies). - -![InferRSI system architecture: inputs, experiment loop and evidence](docs/assets/system-architecture-en.svg) - -[English SVG](docs/assets/system-architecture-en.svg) Β· -[Chinese original](docs/assets/system-architecture-zh-CN.svg) - -The supplied overview and its English translation summarize the intended workflow. -Evidence feeds the next experiment through explicit review and user-selected -constraints; automatic evidence-to-knowledge promotion or RunSpec population is -not implemented. Process-cleanup behavior still needs platform-specific validation. -See the [source audit and proposed remedies](docs/reviews/source-audit-ac70272.md) -for the 34 review points, qualifications and suggested acceptance checks. - -For offline Skill Eval, the [Apps workflow](src/inferrsi/apps/workflows/improvement.py) -combines [Benchmark acceptance rules](src/inferrsi/benchmark/improvement.py) with -[Harness candidate selection](src/inferrsi/harness/improvement.py). The tool adapter -freezes snapshots and carries one budget across probes, a rerun and an independent -holdout. [Controller protocol and limits](docs/harness-improvement.md). - -
-See the implemented modules - -![Implemented module architecture](docs/assets/modular-components-en.svg) - -
- -Existing CLI flags, external JSON metrics, run artifacts and reports are retained. -Python callers using retired module paths must update their imports. -[Compatibility and historical evidence](docs/architecture.md#compatibility-and-historical-evidence) -Β· [Import migration list](CHANGELOG.md). - -
-Find the right directory to extend - -![Module-to-directory map](docs/assets/modular-directory-map-en.svg) - -
- -[Directory map](docs/architecture.md#directory-map) Β· -[Extension guide](docs/architecture.md#where-to-extend) Β· -[Original design diagrams (English / Chinese) β†’](docs/architecture.md#knowledge-and-evidence-ownership) - - - -## πŸ“š Go deeper - -| Start here | What you will find | -| --- | --- | -| [Quickstart](docs/quickstart.md) | Installation, real A/B experiments, and reports | -| [Operation guide](docs/operations.md) | Every CLI/Python operation: entry point, effect class, boundary and example | -| [Bounded skill evaluation](docs/skill-evaluation.md) | Real-client skill/task comparisons, independent outcomes and evidence limits | -| [Offline harness improvement](docs/harness-improvement.md) | One INV/DOM step, quality/cost selection, held-out gate and reproducible controller experiments | -| [Qwen B300](docs/qwen-image-2.1-turbo-b300.md) | Reproduction, profiling, optimization ledger and exact image checks | -| [Qwen Ascend](docs/qwen-image-2.1-turbo-ascend.md) | NPU results, rejected rounds and verification scope | -| [h3-metal lessons](docs/h3-metal-apple-silicon-lessons.md) | MiniMax-H3 on Apple silicon: measurement pitfalls, gains, rejected directions and search-loop lessons | -| [Apple silicon playbook](docs/apple-silicon-playbook.md) | Measured, engine-agnostic lessons for Metal/MLX engines: unified memory, Metal execution and benchmarking a shared Mac | -| [Engineering records](src/inferrsi/knowledge/data/skill_library/records/README.md) | Source revisions, backend applicability and validation limits for contribution, dependency and peer findings | -| [Skill library](src/inferrsi/knowledge/data/skill_library/README.md) | Primary task procedures, systems, playbooks, workflows and casebooks | -| [Source audit](docs/reviews/source-audit-ac70272.md) | 34 review points with proposed remedies, qualifications and validation criteria | -| [Daily learning prompt](docs/daily-learning-prompt.md) | 06:00 Asia/Shanghai L2C + performance workflow and prerequisites | -| [Contributor guide](docs/contributing.md) | Development workflow, evidence contracts, and CI | -| [Engine delivery template](docs/template/README.md) | Copyable engine scaffold, branding, evidence and schema-v1 catalog recipe | - -**Build with us.** Bring a bottleneck, a reproducible optimization, or a tuning lesson. [Open an issue β†’](https://github.com/InferMatrixLab/InferRSI/issues) Use `git commit -s` for contributions. diff --git a/docs/assets/inferrsi-banner.svg b/docs/assets/inferrsi-banner.svg index 26fca7740..7a431dd3f 100644 --- a/docs/assets/inferrsi-banner.svg +++ b/docs/assets/inferrsi-banner.svg @@ -1,38 +1,70 @@ - - - + + InferRSI β€” an agent app to onboard and optimize models + The outer loop improves the engine within one task. The agent proposes changes, InferRSI checks quality and performance, and each round guides the next until InferRSI retests the best candidate in a fresh process. The inner loop improves the optimizer across tasks. Paired runs with knowledge on and off evaluate the value of reuse, and reviewed pull requests can add findings to the library for future tasks. - - - - - - - - - - + + + + + + - - - - - INFERMATRIXLAB / INFERENCE SYSTEMS - InferRSI - Faster inference. Verifiable progress. - BENCHMARKS Β· QUALITY GATES Β· REUSABLE KNOWLEDGE - - - - - - - Profile - Optimize - Validate - Reuse - - Every experiment feeds the next. + + + INFERMATRIXLAB + InferRSI + An agent app to onboard and optimize models + + + + OUTER LOOP + Improve the engine within one task + INNER LOOP + Improve the optimizer across tasks + + + + + + + + Evidence for the next round + + + + + + + + + + + + Propose + Run & checkNΓ— + Keep best + Final retest + + + + + Knowledge library + Skills Β· cases Β· records + Evaluate reuse + Knowledge on vs. off + Run evidence + Patch Β· verdict Β· report + Library updates via reviewed PRs diff --git a/docs/optimization-results.md b/docs/optimization-results.md new file mode 100644 index 000000000..6ea8a360b --- /dev/null +++ b/docs/optimization-results.md @@ -0,0 +1,128 @@ +# Optimization results + +[Back to InferRSI](../README.md#selected-results) Β· [vLLM-Omni model optimization roadmap](https://github.com/InferMatrixLab/InferRSI/issues/416) + +The tables show measured improvements of at least 20%. Hardware, workloads and +baseline revisions differ across comparisons. Quality and acceptance status are +reported for each result. + +## Text serving + +### DeepSeek V4.1 Flash W4A8 on Ascend A3 + +The project ran DeepSeek V4.1 Flash 1017 W4A8 on an Ascend A3 SuperPod with +disaggregated prefill and decode (1P1D, 8 physical cards per side). AISBench +generated load and InferRSI rechecked throughput at each scenario's latency +limit. [Project report, configuration and reproduction](https://github.com/jiaran-king/dsv4.1-optimize). + +| Scenario | Per-card throughput metric | Latency limit | Baseline β†’ optimized (tokens/s) | Measured change | +| --- | --- | --- | --- | --- | +| 19 Β· Prefill | Input throughout the run | Mean TTFT < 10 s | 68,239 β†’ 105,397 | **+54.45%** | +| 21 Β· Decode, C1152 | Steady-state output | Mean TPOT < 50 ms | 1,711.98 β†’ 2,330.28 | **+36.12%** | +| 23 Β· End-to-end, steady state | Decode output | Mean TPOT < 21 ms | 1,132.9 β†’ 1,368.5 | **+20.80%** | + +Cases 19 and 23 use medians from two runs. Case 21 uses one recheck of the +retained configuration, which includes a SiLU safeguard. TTFT is time to first +token. TPOT is time per output token. + +### Historical DeepSeek V4.1 Flash W8A8 pilot + +The A3 / 910C pilot used 16 dies, TP4 / DP4 / EP16, DSpark-5 and an +8K-input / 1K-output workload. The slide deck reported the following single-run +observations against the earlier control in [issue #329](https://github.com/InferMatrixLab/InferRSI/issues/329). + +| Concurrency | Earlier control β†’ reported pilot TPOT (ms/token) | Measured change | +| --- | --- | --- | +| 1 | 93.86 β†’ 30.02 | **βˆ’68.0%** | +| 4 | 97.36 β†’ 24.98 | **βˆ’74.3%** | +| 16 | 102.33 β†’ 36.55 | **βˆ’64.3%** | + +Only C4 reached 25 ms/token in the reported run. The follow-up benchmark uses +its own calibrated baseline. The campaign stopped with an **INCONCLUSIVE** +verdict. [Follow-up #353](https://github.com/InferMatrixLab/InferRSI/issues/353). + +## Image, video and speech in vLLM-Omni + +InferRSI campaigns and upstream experiments cover image, video and speech models. +Each result applies to its linked workload and measured revisions. Some results +are reported directly by upstream contributors, with validation scoped to their benchmarks. + +| Model | Hardware | Optimization approach | Measured gains | Validation and limits | Evidence | vLLM-Omni PR | +| --- | --- | --- | --- | --- | --- | --- | +| **Qwen-Image-2.1-Turbo** | 4Γ— B300 |
  • Text-conditioning reuse
  • BF16 operator fusion
  • Symmetric-memory Ulysses with batched QKV exchange
| **384.16 β†’ 266.60 ms (βˆ’30.60%)**
Warm HTTP end-to-end latency, repeated prompts | Historical InferRSI campaign. Latencies are medians of five trial means from alternating AB/BA groups. All 44 RGBA images were pixel-identical. | [Campaign #3](https://github.com/InferMatrixLab/InferRSI/issues/3) Β· [Pinned results](qwen-image-2.1-turbo-b300.md) | [#8739](https://github.com/vllm-project/vllm-omni/pull/8739) (merged) | +| **Qwen-Image-2.1-Turbo** | 1Γ— Ascend 910B1 |
  • Conditioning cache for repeated prompts
  • Shared modulation once per denoising step
  • Native interleaved NPU RoPE with frequency-table reuse
| **2,921.23 β†’ 2,272.83 ms (βˆ’22.20%)** offline generation latency
Warm repeated prompts | Upstream parent-to-merge reference. InferRSI imported the techniques as knowledge. 18/18 RGBA images were pixel-identical. The 30% target was unmet. Both arms aborted during teardown, leaving lifecycle `INCONCLUSIVE`. | [Final-merge measurements](https://github.com/vllm-project/vllm-omni/pull/8729#:~:text=Final%2Dmerge%20performance%20and%20image%20comparison) Β· [Earlier Ascend cohorts](qwen-image-2.1-turbo-ascend.md) | [#8729](https://github.com/vllm-project/vllm-omni/pull/8729) (merged) | +| **MiniMax-H3 Ref2VA** | 8Γ— Ascend 910B1 |
  • Lossy Cache-DiT profile tuning
| **407.27 β†’ 108.12 s (βˆ’73.45%)**
5 s video, HTTP end-to-end vs. dense inference | All 14 frozen quality comparisons passed. | [Campaign #293](https://github.com/InferMatrixLab/InferRSI/issues/293) Β· [Pinned results](../evidence/h3-ref2va-ascend-293/results/final-5s/README.md) | [#9698](https://github.com/vllm-project/vllm-omni/pull/9698) (draft) | +| **MiniMax-H3 Ref2VA** | 4Γ— Ascend 950PR (~86 GB per card) |
  • USP4 + MXFP8
  • RainFusion + Cache-DiT
| **701.95 β†’ 166.17 s (βˆ’76.3%, 4.22Γ—)**
5 s clip denoising time vs. the in-run dense control | Five retained historical A/B slots. Performance and quality qualification remained inconclusive. For quality, the recipe reports sharpness, flicker, and luma ratios of 0.924 / 0.950 / 0.984 against a dense USP4 + MXFP8 reference. | [Archive record #520](https://github.com/InferMatrixLab/InferRSI/issues/520) Β· [Measured repeats](https://github.com/vllm-project/vllm-omni/pull/9955#:~:text=in%2Drun%20dense%20median) | [#9955](https://github.com/vllm-project/vllm-omni/pull/9955) (merged recipe) | +| **MiniMax-H3-Image** | 4Γ— Ascend 910B1 |
  • Opt-in VAE tile graphs
  • Approximate encoder/decoder tile batching
| **16.89 β†’ 10.82 s (βˆ’35.9%)** Β· 4736Γ—3552 generation
**4.18 β†’ 2.67 s (βˆ’36.1%)** Β· 1024Γ—1024 generation | Single-launch HTTP medians vs. the same-layout SP4 control. All 30 numerical image screens passed across generation and editing. Latency targets were unmet and final qualification remains `INCONCLUSIVE`. | [Campaign #390](https://github.com/InferMatrixLab/InferRSI/issues/390) Β· [Evidence PR #394](https://github.com/InferMatrixLab/InferRSI/pull/394) | [#10184](https://github.com/vllm-project/vllm-omni/pull/10184) (merged) | +| **MiniMax-H3-Image** | 2Γ— RTX 3090 |
  • Rank-local layerwise offload
  • Selected DiT weights kept resident
  • AdaLN caching
  • Request-local text refinement reuse
| **36.62 β†’ 14.59 s (βˆ’60.16%)**
Warm mean HTTP delivery latency | One upstream A/B of complete deployment profiles. All nine image pairs had identical decoded RGB pixels, including warmups. Repeated qualification remains incomplete. | [Paired deployment benchmark](https://github.com/vllm-project/vllm-omni/pull/10243#:~:text=One%20paired%20deployment%20comparison) | [#10243](https://github.com/vllm-project/vllm-omni/pull/10243) (open) | +| **MiniCPM-o 4.5** | 1Γ— L20X | Seed-TTS runtime and deployment changes | **11.46 β†’ 22.99 requests/s (+100.6%)** and **0.65548 β†’ 0.32300 mean RTF (βˆ’50.72%)** at C32
**10.62 β†’ 18.51 requests/s (+74.29%)** and **0.34287 β†’ 0.19744 mean RTF (βˆ’42.42%)** at C16
**2.94 β†’ 3.66 requests/s (+24.49%)** at C1 | Five retained fresh servers per arm, 12,800 successful requests. The independent-engine target was outside this comparison. Formal speech-quality non-inferiority remains unestablished. | [Campaign #431](https://github.com/InferMatrixLab/InferRSI/issues/431) | [#10331](https://github.com/vllm-project/vllm-omni/pull/10331) (merged) | +| **MiniCPM-o 4.5** | 1Γ— L20X |
  • Packed conditioning on a tuned configuration
  • TensorRT speaker extraction + CUDA MPS (separate comparison)
| **βˆ’27.69%** median TTFA
Packed conditioning, C32, same 50-frame window
**+20.5%** at C16 Β· **+25.4%** at C32
TensorRT + MPS throughput | The packed-conditioning candidate missed the exploratory WER margin against the tuning reference at the 50-frame setting. The separate TensorRT + MPS comparison had speech-quality verdict `FAIL`. | [Campaign #49](https://github.com/InferMatrixLab/InferRSI/issues/49) | [#9218](https://github.com/vllm-project/vllm-omni/pull/9218), [#9188](https://github.com/vllm-project/vllm-omni/pull/9188) (both merged) | +| **LFM3-Omni-2B** | 1Γ— Ascend 910B1 |
  • Depth graphs
  • Grouped greedy sampling
| **0.285 β†’ 0.593 requests/s (+108.4%)**
Upstream offline Omni API A/B, C4, temperature 0 | Retained audio waveforms matched bit-for-bit within each tested temperature. | [Offline A/B benchmark](https://github.com/vllm-project/vllm-omni/pull/9827#:~:text=Performance%20results) | [#9827](https://github.com/vllm-project/vllm-omni/pull/9827) (merged) | +| **Drama 3** | 2Γ— Ascend 910B1 | Full graph replay for steady-state decode | **17.31 β†’ 22.40 / 22.91 frames/s (+31%)** and **1.237 β†’ 0.962 / 0.947 mean RTF (βˆ’23%)** at C1 | Rounded changes use the mean of the two candidate runs. The real-time factor targets were unmet. | [Campaign #132](https://github.com/InferMatrixLab/InferRSI/issues/132) | [#9308](https://github.com/vllm-project/vllm-omni/pull/9308) (merged) | +| **Qwen3-TTS 1.7B** | 1Γ— Ascend 910B1 | Codec and predictor optimizations | **1.869 β†’ 4.822 audio s/s (+158%)** at C1
**16.463 β†’ 38.755 (+135%)** at C16
**21.245 β†’ 52.091 (+145%)** at C32 | Historical seven-path profile vs. fresh same-card no-PR control. Mandarin speaker similarity and some ASR results regressed. Upstream reference. | [vLLM-Omni roadmap #416](https://github.com/InferMatrixLab/InferRSI/issues/416) | [#9068](https://github.com/vllm-project/vllm-omni/pull/9068) (merged) | + +**Earlier Qwen-Image-2.1-Turbo Ascend measurements.** On one Ascend 910B1 at +1280Γ—704, BF16, eager, and eight native steps, the author reported a 3111.5 ms +baseline. Conditioning reuse with interleaved RoPE reached **2459.6 ms (βˆ’21.0%)** +with pixel-identical RGBA. Adding the opt-in VAE fastpath reached **2425.5 ms +(βˆ’22.0%)** and passed the reported quality gate with changed pixels. Each mean +covers three requests after two warmups. InferRSI verification covered the +import, so acceptance remains `INCONCLUSIVE`. The 30% target was unmet. The B300 +campaign and this cohort used different hardware and measurement protocols. +[Earlier Ascend report](qwen-image-2.1-turbo-ascend.md) + +### Historical MiniMax-H3 Ref2VA step-reduction measurements on 950PR + +The author reported these 5-second workloads on four Ascend 950PR cards +relative to their corresponding 50-step dense controls. Each sampling +configuration used three seeds and eleven prompts. Both performance and quality +qualification remained **INCONCLUSIVE**. One seed in the 10-step dense +configuration lost 9.17% subject consistency. The 10-step Cache-DiT route +combines fewer denoising steps and lossy caching, so its speedup describes the +whole workload change. [Record #520](https://github.com/InferMatrixLab/InferRSI/issues/520) Β· +[Evidence index #527](https://github.com/InferMatrixLab/InferRSI/pull/527). + +| Configuration | Speedup | Relative time reduction | +| --- | --- | --- | +| 10-step dense | 5.04Γ— | **βˆ’80.2%** | +| 10-step + Cache-DiT 0.12 | 7.37Γ— | **βˆ’86.4%** | +| 11-step dense | 4.74Γ— | **βˆ’78.9%** | +| 12-step dense | 4.34Γ— | **βˆ’77.0%** | + +TTFA is time to first audio. RTF is real-time factor. WER is word error rate. + +Upstream PR statuses checked on 2026-11-20. To refresh a row, substitute its PR +number in `gh pr view 10184 --repo vllm-project/vllm-omni --json state,isDraft`. + +## Standalone engine benchmarks + +### Standalone video/audio engines + +[h3-metal](https://github.com/zuiho-kai/h3-metal) provides Metal-native FastH3 and MiniMax-H3 text-to-video/audio inference on Apple GPUs. + +Author-reported results on **Apple M5 Max (40-core GPU), 128 GB, macOS 27**, at **768Γ—448, 49 frames, seed 0**, using real weights with one engine running at a time. + +| Engine | Device | Steps | Cold | Warm | vs. MLX (cold / warm) | vs. h3.c (cold / warm) | +| --- | --- | --- | --- | --- | --- | --- | +| `PipeNetwork/minimax-h3-mlx` (4-bit) | Apple M5 Max (40-core GPU) | 20 | 153.1 s | 134.6 s | 1.00Γ— / 1.00Γ— | 0.44Γ— / 0.54Γ— | +| `antirez/h3.c`, default preset | Apple M5 Max (40-core GPU) | 20 | 67.6 s | 73.3 s | 2.26Γ— / 1.84Γ— | 1.00Γ— / 1.00Γ— | +| **h3-metal**, original H3 | Apple M5 Max (40-core GPU) | 20 | **54.29 s** | **29.89 s** | **2.82Γ— / 4.50Γ—** | **β€” / 2.45Γ—** | +| **h3-metal**, FastH3 4-step + block-sparse attention | Apple M5 Max (40-core GPU) | 4 ΒΉ | **48.23 s** | **18.20 s** | **3.17Γ— / 7.40Γ—** | **1.40Γ— / 4.03Γ—** | + +Cold includes the first clip's weight loading, text encoding and preparation; warm reuses loaded weights and prepared state. + +ΒΉ FastH3 uses a distilled 4-step checkpoint, unlike the 20-step configurations. Warm values average clips 2–4 for the 20-step rows and two A/B/B/A warm runs for FastH3. + +[Author's report and reproduction β†’](https://github.com/zuiho-kai/h3-metal/blob/53b7fa5d88e2497ab427ab9c7155535f7e6e6817/README.md#performance). These external results were not rerun in InferRSI. + +### Standalone speech engines + +Concurrency 1 (C1) results reported by each project. Values are **engine / vLLM-Omni / SGLang-Omni**; compare within a row, since workloads and audio-frame definitions differ. + +| Project | Device | Throughput (frames/s) ↑ | TTFA p50 (ms) ↓ | Report and status | +| --- | --- | --- | --- | --- | +| [Drama 3](https://github.com/InferMatrixLab/drama-3-single-b300-engine) | 1Γ— B300 | **309.17** / 195.89 / 169.80 | **32.83** / 47.29 / 79.38 | [3-run medians](https://github.com/InferMatrixLab/drama-3-single-b300-engine/blob/58a1882a4d941f8d21bfce6be37d3d5e7c9fde80/docs/current-results.md); overall target unmet | +| [Qwen3-TTS](https://github.com/InferMatrixLab/qwen3-tts-rsi-npu-engine) | 1Γ— Ascend 910B1 | **61.12** / 12.71 / 4.31 | **50.41** / 263.51 / 439.72 | [3-run means](https://github.com/InferMatrixLab/qwen3-tts-rsi-npu-engine/blob/a29f8193b557c415175d70919bdfb9bd3dcd51fc/evidence/campaign-20261017/same-card-performance.md); source campaign accepted | +| [MiniCPM-o 4.5](https://github.com/InferMatrixLab/minicpm-o-4.5-single-gpu-engine) | 1Γ— L20X | **397.43** / 187.52 / 278.72 | **126.2** / 295.3 / 381.2 | [5-run means](https://github.com/InferMatrixLab/minicpm-o-4.5-single-gpu-engine/blob/a4f4778f312791c2d190b48e521141887ed160c8/docs/current-results.md); target unmet, quality unqualified | + +MiniCPM's SGLang baseline returns a complete waveform, so its TTFA equals end-to-end latency. These external results were not rerun in InferRSI. From c242ac5e7f7b3db09201a82021b58937049de4b2 Mon Sep 17 00:00:00 2001 From: Dong1017 <55723162+Dong1017@users.noreply.github.com> Date: Fri, 20 Nov 2026 20:37:43 +0800 Subject: [PATCH 4/7] docs: add emoji to README navigation Agent-assisted-by: Codex (author) Signed-off-by: Dong1017 <55723162+Dong1017@users.noreply.github.com> --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index b57b7f13b..eb14e2e6d 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ ![InferRSI β€” an agent app to onboard and optimize models. The outer loop improves the engine and the inner loop improves the optimizer.](docs/assets/inferrsi-banner.svg) -[Quickstart](#quickstart) Β· [How It Works](#how-it-works) Β· [Selected Results](#selected-results) Β· [Docs](#documentation) Β· [Contributing](#contributing) +[πŸš€ Quickstart](#quickstart) Β· [πŸ” How It Works](#how-it-works) Β· [πŸ“Š Selected Results](#selected-results) Β· [πŸ“š Docs](#documentation) Β· [🀝 Contributing](#contributing) From 43d313545eb40a26d891a0720054963c78159784 Mon Sep 17 00:00:00 2001 From: Dong1017 <55723162+Dong1017@users.noreply.github.com> Date: Fri, 20 Nov 2026 22:10:58 +0800 Subject: [PATCH 5/7] docs: add section emoji, tighten result tables, address review - Add emoji to section headings and keep every previous anchor id so existing links into the README still resolve. - Merge the Baseline column into the measured-change column, taking the result tables from five columns to four for narrow screens, and shorten the validation cells. - LFM3-Omni-2B: report +108.1%, which is what the listed 0.285 -> 0.593 req/s endpoints give; the details page notes the upstream +108.4% comes from unrounded means. - DeepSeek W4A8 prefill: "Input throughput" instead of "Input throughout the run". - Restore the three below-20% vLLM-Omni campaigns to the details page in their own section, with their draft and merged labels intact, and stop describing the page as only showing 20%+ tables. - Point the six-run DeepSeek baseline at the comment that states 39.28 / 44.07 / 56.86 ms/token. - Dashboard example: use `inferrsi dashboard --input .` and its dashboard.html default, since `run` already writes report.html and the exporter refuses to overwrite a document it did not generate. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Dong1017 <55723162+Dong1017@users.noreply.github.com> --- README.md | 128 ++++++++++++++++++++--------------- docs/optimization-results.md | 28 ++++++-- 2 files changed, 96 insertions(+), 60 deletions(-) diff --git a/README.md b/README.md index eb14e2e6d..8cc0afa8a 100644 --- a/README.md +++ b/README.md @@ -10,9 +10,11 @@ InferRSI helps coding agents and engineers profile inference workloads, test optimizations against a baseline, and verify output quality before adopting a change. Bring an engine checkout, a workload, and your quality checks. -## Quickstart + -**Use the CLI.** Install from source and list the packaged benchmarks on a +## πŸš€ Quickstart + +**πŸ–₯️ Use the CLI.** Install from source and list the packaged benchmarks on a development machine with Python 3.10+. ```bash @@ -34,15 +36,17 @@ Setup and task discovery run without starting a benchmark. For a target host, select its backend (`cpu`, `cuda`, `rocm`, `ascend`, or `mlx`). [Install guide](docs/quickstart.md#install-and-smoke) Β· [Release wheels and checksums](https://github.com/InferMatrixLab/InferRSI/releases) -**Use a coding agent.** Install the plugin and ask it to check the target +**πŸ€– Use a coding agent.** Install the plugin and ask it to check the target checkout, workload, and quality criteria before starting an experiment. [Codex](docs/codex-plugin.md#install-from-the-release-zip) Β· [DeepSeek Harness](docs/dsh-plugin.md) Β· [WorkSwarm](docs/workswarm-plugin.md#download-and-install) -**Run your first experiment.** [Try a packaged benchmark](docs/optimization-benchmarks.md#start-the-complete-workflow) Β· [Bring your own workload](docs/quickstart.md#real-remote-experiment) +**πŸ§ͺ Run your first experiment.** [Try a packaged benchmark](docs/optimization-benchmarks.md#start-the-complete-workflow) Β· [Bring your own workload](docs/quickstart.md#real-remote-experiment) -## How It Works + + +## πŸ” How It Works **Outer loop.** To improve the engine, the agent profiles the workload and proposes a change. InferRSI checks quality and measures the candidate against @@ -65,7 +69,7 @@ result views. **Review a run in the dashboard.** Compare baseline and candidate profiles and trace each experiment back to its evidence. From a recorded run directory, -generate a local report with `inferrsi dashboard --input . --output report.html`. +open it with `inferrsi dashboard --input .`, which writes `dashboard.html` beside the run. [Dashboard guide](docs/dashboard.md#generate-and-open) Β· [Download the interactive example](docs/examples/dashboard-profile-review.html)

@@ -78,76 +82,94 @@ generate a local report with `inferrsi dashboard --input . --output report.html` -## Selected Results + + +## πŸ“Š Selected Results Model bring-up and measured optimizations across text, image, video, and speech. Performance tables highlight metrics that improved by at least 20% against the listed baseline. The evidence column gives the scope and validation status. +πŸ”₯ marks the largest gain in its table, βœ… a row whose quality checks passed, and +⚠️ a failed or unqualified check, or an inconclusive campaign. -### Model onboarding + + +### 🧩 Model onboarding | Model | Engine and hardware | Milestone | Evidence | | --- | --- | --- | --- | -| **DeepSeek V4.1 Flash** | vLLM-Ascend Β· 8Γ— Ascend 910C | W8A8 service ran the 8K-input / 1K-output workload. Six independent service runs established a baseline of **39.28 / 44.07 / 56.86 ms/token** at concurrency 1 / 4 / 16. The 25 ms/token target remains unmet. | [Bring-up and baseline](https://github.com/InferMatrixLab/InferRSI/issues/353#:~:text=9be3c22ab5dade39c0d33d49540ae021ff74bbd1) | -| **Qwen3.8-27B** | Standalone nano-vLLM-derived engine Β· 1Γ— H20 | The FP8 text service passed scoped HTTP/SSE lifecycle checks. Model-quality and performance qualification remained incomplete. | [Service bring-up](https://github.com/InferMatrixLab/InferRSI/issues/57) Β· [Performance campaign](https://github.com/InferMatrixLab/InferRSI/issues/67) | -| **LFM3-Omni-2B** | vLLM-Omni Β· 1Γ— Ascend 910B1 | Two real-weight serving tests passed. A functional eager adapter provided a baseline for the offline audio benchmark. | [Serving checks and A/B benchmark](https://github.com/vllm-project/vllm-omni/pull/9827) | - -### Text serving - -| Model | Measured change | Metric and hardware | Baseline | Validation and evidence | -| --- | --- | --- | --- | --- | -| **DeepSeek V4.1 Flash** | **+54.45% / +36.12% / +20.80%** | Per-card throughput for prefill / decode / end-to-end decode output Β· Ascend A3, W4A8, 1P1D | 68,239 / 1,711.98 / 1,132.9 tokens/s per card | InferRSI recheck at each scenario's latency limit. The decode result was rechecked once. [Workloads and evidence](https://github.com/jiaran-king/dsv4.1-optimize) | - -### Image and video generation - -| Model | Measured change | Metric and hardware | Baseline | Validation and evidence | -| --- | --- | --- | --- | --- | -| **MiniMax-H3 Ref2VA** | **βˆ’73.45%** | 5 s video, HTTP latency Β· 8Γ— Ascend 910B1 | Dense inference Β· 407.27 s | **3.77Γ—** with lossy Cache-DiT. All 14 quality comparisons passed. [Campaign](https://github.com/InferMatrixLab/InferRSI/issues/293) Β· [Benchmark](evidence/h3-ref2va-ascend-293/results/final-5s/README.md) | -| **MiniMax-H3 Ref2VA** | **βˆ’76.3%** | 5 s video, 50-step denoising time Β· 4Γ— Ascend 950PR | In-run dense control Β· 701.95 s | **4.22Γ—** in five retained A/B slots. Historical record with inconclusive performance and quality qualification. [Archive record](https://github.com/InferMatrixLab/InferRSI/issues/520) Β· [Recipe benchmark](https://github.com/vllm-project/vllm-omni/pull/9955#:~:text=in%2Drun%20dense%20median) | -| **MiniMax-H3-Image** | **βˆ’60.16%** | Warm HTTP latency Β· 2Γ— RTX 3090 | Original layer offload Β· 36.62 s | One paired comparison with 9 pixel-identical outputs. [Upstream results](https://github.com/vllm-project/vllm-omni/pull/10243#:~:text=One%20paired%20deployment%20comparison) | -| **MiniMax-H3-Image** | **βˆ’35.9% / βˆ’36.1%** | HTTP generation latency at 4736Γ—3552 / 1024Γ—1024 Β· 4Γ— Ascend 910B1 | Same-layout SP4 control Β· 16.89 / 4.18 s | All 30 numerical checks passed across the tested workloads. Latency targets were unmet. [Benchmark](https://github.com/InferMatrixLab/InferRSI/pull/394) | -| **Qwen-Image-2.1-Turbo** | **βˆ’30.60%** | Warm HTTP latency Β· 4Γ— B300 | Pinned control Β· 384.16 ms | 44 pixel-identical outputs. [InferRSI results](docs/qwen-image-2.1-turbo-b300.md) | -| **Qwen-Image-2.1-Turbo** | **βˆ’22.20%** | Warm offline generation latency Β· 1Γ— Ascend 910B1 | Upstream parent Β· 2,921.23 ms | Upstream reference with 18 pixel-identical outputs. Teardown validation was inconclusive. [Upstream results](https://github.com/vllm-project/vllm-omni/pull/8729#:~:text=Final%2Dmerge%20performance%20and%20image%20comparison) | - -### Speech and multimodal serving - -| Model | Measured change | Metric and hardware | Baseline | Validation and evidence | -| --- | --- | --- | --- | --- | -| **LFM3-Omni-2B** | **+108.4%** | Offline greedy request throughput Β· 1Γ— Ascend 910B1, C4 | Functional eager adapter Β· 0.285 req/s | Retained audio matched bit-for-bit within each tested temperature. [Upstream results](https://github.com/vllm-project/vllm-omni/pull/9827#:~:text=Performance%20results) | -| **MiniCPM-o 4.5** | **+100.6% / βˆ’50.72%** | Request throughput / mean real-time factor Β· 1Γ— L20X, C32 | Pinned upstream main Β· 11.46 req/s / 0.65548 RTF | Five matched runs per arm. Formal speech-quality non-inferiority remains unestablished. [Campaign](https://github.com/InferMatrixLab/InferRSI/issues/431) Β· [Benchmark](https://github.com/vllm-project/vllm-omni/pull/10331) | -| **MiniCPM-o 4.5** | **βˆ’27.69%** | Median time to first audio Β· 1Γ— L20X, C32 | Tuned 50-frame deployment before packed conditioning | Packed conditioning missed the exploratory word error rate margin. [Benchmark](https://github.com/vllm-project/vllm-omni/pull/9218) | -| **MiniCPM-o 4.5** | **+20.5% / +25.4%** | Throughput at C16 / C32 Β· 1Γ— L20X | Pinned upstream main before TensorRT + MPS | The speech quality check failed. [Benchmark](https://github.com/vllm-project/vllm-omni/pull/9188) | -| **Qwen3-TTS 1.7B** | **+158%** | Streaming audio throughput Β· 1Γ— Ascend 910B1, C1 | Fresh no-PR control Β· 1.869 audio s/s | Historical upstream reference. Mandarin speaker similarity and some ASR scores regressed. [Upstream benchmark](https://github.com/vllm-project/vllm-omni/pull/9068) | -| **Drama 3** | **+31% / βˆ’23%** | Throughput / mean real-time factor Β· 2Γ— Ascend 910B1, C1 | Piecewise graphs Β· 17.31 frames/s / 1.237 RTF | Two runs per arm. The real-time factor targets were unmet. [Campaign](https://github.com/InferMatrixLab/InferRSI/issues/132) Β· [Benchmark](https://github.com/vllm-project/vllm-omni/pull/9308) | +| **DeepSeek V4.1 Flash** | vLLM-Ascend Β· 8Γ— Ascend 910C | W8A8 service on the 8K-input / 1K-output workload. Six independent runs gave **39.28 / 44.07 / 56.86 ms/token** at C1 / C4 / C16; the 25 ms/token target remains unmet. | [Bring-up and baseline](https://github.com/InferMatrixLab/InferRSI/issues/353#issuecomment-6641425447) | +| **Qwen3.8-27B** | Standalone nano-vLLM-derived engine Β· 1Γ— H20 | FP8 service passed scoped HTTP/SSE lifecycle checks; quality and performance qualification incomplete. | [Service bring-up](https://github.com/InferMatrixLab/InferRSI/issues/57) Β· [Performance campaign](https://github.com/InferMatrixLab/InferRSI/issues/67) | +| **LFM3-Omni-2B** | vLLM-Omni Β· 1Γ— Ascend 910B1 | βœ… Two real-weight serving tests passed; a functional eager adapter served as the offline audio baseline. | [Serving checks and A/B benchmark](https://github.com/vllm-project/vllm-omni/pull/9827) | + + + +### πŸ’¬ Text serving + +| Model | Measured change and baseline | Metric and hardware | Validation and evidence | +| --- | --- | --- | --- | +| **DeepSeek V4.1 Flash** | **+54.45% / +36.12% / +20.80%**
Baseline: 68,239 / 1,711.98 / 1,132.9 tokens/s per card | Per-card throughput, prefill / decode / end-to-end Β· Ascend A3, W4A8, 1P1D | βœ… Rechecked at each scenario's latency limit; decode rechecked once. [Workloads and evidence](https://github.com/jiaran-king/dsv4.1-optimize) | + + + +### 🎨 Image and video generation + +| Model | Measured change and baseline | Metric and hardware | Validation and evidence | +| --- | --- | --- | --- | +| **MiniMax-H3 Ref2VA** | **βˆ’73.45%**
Baseline: Dense inference Β· 407.27 s | 5 s video, HTTP latency Β· 8Γ— Ascend 910B1 | **πŸ”₯ 3.77Γ—** with lossy Cache-DiT. βœ… 14 / 14 quality comparisons passed. [Campaign](https://github.com/InferMatrixLab/InferRSI/issues/293) Β· [Benchmark](evidence/h3-ref2va-ascend-293/results/final-5s/README.md) | +| **MiniMax-H3 Ref2VA** | **βˆ’76.3%**
Baseline: In-run dense control Β· 701.95 s | 5 s video, 50-step denoising time Β· 4Γ— Ascend 950PR | **πŸ”₯ 4.22Γ—** in five retained A/B slots. ⚠️ Historical; performance and quality inconclusive. [Archive record](https://github.com/InferMatrixLab/InferRSI/issues/520) Β· [Recipe benchmark](https://github.com/vllm-project/vllm-omni/pull/9955#:~:text=in%2Drun%20dense%20median) | +| **MiniMax-H3-Image** | **βˆ’60.16%**
Baseline: Original layer offload Β· 36.62 s | Warm HTTP latency Β· 2Γ— RTX 3090 | **πŸ”₯** One paired comparison; βœ… 9 pixel-identical outputs. [Upstream results](https://github.com/vllm-project/vllm-omni/pull/10243#:~:text=One%20paired%20deployment%20comparison) | +| **MiniMax-H3-Image** | **βˆ’35.9% / βˆ’36.1%**
Baseline: Same-layout SP4 control Β· 16.89 / 4.18 s | HTTP generation latency at 4736Γ—3552 / 1024Γ—1024 Β· 4Γ— Ascend 910B1 | βœ… 30 / 30 numerical checks passed. Latency targets unmet. [Benchmark](https://github.com/InferMatrixLab/InferRSI/pull/394) | +| **Qwen-Image-2.1-Turbo** | **βˆ’30.60%**
Baseline: Pinned control Β· 384.16 ms | Warm HTTP latency Β· 4Γ— B300 | βœ… 44 pixel-identical outputs. [InferRSI results](docs/qwen-image-2.1-turbo-b300.md) | +| **Qwen-Image-2.1-Turbo** | **βˆ’22.20%**
Baseline: Upstream parent Β· 2,921.23 ms | Warm offline generation latency Β· 1Γ— Ascend 910B1 | ⚠️ Upstream reference; 18 pixel-identical outputs. Teardown inconclusive. [Upstream results](https://github.com/vllm-project/vllm-omni/pull/8729#:~:text=Final%2Dmerge%20performance%20and%20image%20comparison) | + + + +### πŸ”Š Speech and multimodal serving + +| Model | Measured change and baseline | Metric and hardware | Validation and evidence | +| --- | --- | --- | --- | +| **LFM3-Omni-2B** | **+108.1%**
Baseline: Functional eager adapter Β· 0.285 req/s | Offline greedy request throughput Β· 1Γ— Ascend 910B1, C4 | **πŸ”₯** βœ… Retained audio bit-identical at every tested temperature. [Upstream results](https://github.com/vllm-project/vllm-omni/pull/9827#:~:text=Performance%20results) | +| **MiniCPM-o 4.5** | **+100.6% / βˆ’50.72%**
Baseline: Pinned upstream main Β· 11.46 req/s / 0.65548 RTF | Request throughput / mean real-time factor Β· 1Γ— L20X, C32 | ⚠️ Five matched runs per arm; speech-quality non-inferiority unestablished. [Campaign](https://github.com/InferMatrixLab/InferRSI/issues/431) Β· [Benchmark](https://github.com/vllm-project/vllm-omni/pull/10331) | +| **MiniCPM-o 4.5** | **βˆ’27.69%**
Baseline: Tuned 50-frame deployment before packed conditioning | Median time to first audio Β· 1Γ— L20X, C32 | ⚠️ Missed the exploratory word-error-rate margin. [Benchmark](https://github.com/vllm-project/vllm-omni/pull/9218) | +| **MiniCPM-o 4.5** | **+20.5% / +25.4%**
Baseline: Pinned upstream main before TensorRT + MPS | Throughput at C16 / C32 Β· 1Γ— L20X | ⚠️ Speech quality check failed. [Benchmark](https://github.com/vllm-project/vllm-omni/pull/9188) | +| **Qwen3-TTS 1.7B** | **+158%**
Baseline: Fresh no-PR control Β· 1.869 audio s/s | Streaming audio throughput Β· 1Γ— Ascend 910B1, C1 | **πŸ”₯** ⚠️ Historical upstream reference; Mandarin speaker similarity and some ASR scores regressed. [Upstream benchmark](https://github.com/vllm-project/vllm-omni/pull/9068) | +| **Drama 3** | **+31% / βˆ’23%**
Baseline: Piecewise graphs Β· 17.31 frames/s / 1.237 RTF | Throughput / mean real-time factor Β· 2Γ— Ascend 910B1, C1 | Two runs per arm; real-time-factor targets unmet. [Campaign](https://github.com/InferMatrixLab/InferRSI/issues/132) Β· [Benchmark](https://github.com/vllm-project/vllm-omni/pull/9308) | -### Standalone engines + + +### βš™οΈ Standalone engines Engine projects report these comparisons. Speech benchmarks use one concurrent request per engine and measure audio frames per second and median time to first audio. -| Engine | Measured change | Metric and hardware | Baseline | Benchmark scope and evidence | -| --- | --- | --- | --- | --- | -| **Qwen3-TTS** | **+381% / βˆ’81%** | Audio throughput / time to first audio Β· 1Γ— Ascend 910B1 | vLLM-Omni Β· 12.71 frames/s / 263.51 ms | Three independent runs per engine. Candidate audio and contract checks passed. [Same-card benchmark](https://github.com/InferMatrixLab/qwen3-tts-rsi-npu-engine/blob/a29f8193b557c415175d70919bdfb9bd3dcd51fc/evidence/campaign-20261017/same-card-performance.md) | -| **MiniCPM-o 4.5** | **+112% / βˆ’57%** | Audio throughput / time to first audio Β· 1Γ— L20X | vLLM-Omni Β· 187.52 frames/s / 295.3 ms | Performance target unmet and quality unqualified. [Same-card benchmark](https://github.com/InferMatrixLab/minicpm-o-4.5-single-gpu-engine/blob/a4f4778f312791c2d190b48e521141887ed160c8/docs/current-results.md) | -| **Drama 3** | **+58% / βˆ’31%** | Audio throughput / time to first audio Β· 1Γ— B300 | vLLM-Omni Β· 195.89 frames/s / 47.29 ms | Three-run medians. The overall target was unmet. [Same-card benchmark](https://github.com/InferMatrixLab/drama-3-single-b300-engine/blob/58a1882a4d941f8d21bfce6be37d3d5e7c9fde80/docs/current-results.md) | -| **h3-metal** | **βˆ’59%** | Warm video generation latency Β· Apple M5 Max | h3.c Β· 73.3 s | Both engines used the original MiniMax-H3 at 20 steps. [Author benchmark](https://github.com/zuiho-kai/h3-metal/blob/53b7fa5d88e2497ab427ab9c7155535f7e6e6817/README.md#performance) | +| Engine | Measured change and baseline | Metric and hardware | Benchmark scope and evidence | +| --- | --- | --- | --- | +| **Qwen3-TTS** | **+381% / βˆ’81%**
Baseline: vLLM-Omni Β· 12.71 frames/s / 263.51 ms | Audio throughput / time to first audio Β· 1Γ— Ascend 910B1 | **πŸ”₯** βœ… Three runs per engine; audio and contract checks passed. [Same-card benchmark](https://github.com/InferMatrixLab/qwen3-tts-rsi-npu-engine/blob/a29f8193b557c415175d70919bdfb9bd3dcd51fc/evidence/campaign-20261017/same-card-performance.md) | +| **MiniCPM-o 4.5** | **+112% / βˆ’57%**
Baseline: vLLM-Omni Β· 187.52 frames/s / 295.3 ms | Audio throughput / time to first audio Β· 1Γ— L20X | ⚠️ Target unmet; quality unqualified. [Same-card benchmark](https://github.com/InferMatrixLab/minicpm-o-4.5-single-gpu-engine/blob/a4f4778f312791c2d190b48e521141887ed160c8/docs/current-results.md) | +| **Drama 3** | **+58% / βˆ’31%**
Baseline: vLLM-Omni Β· 195.89 frames/s / 47.29 ms | Audio throughput / time to first audio Β· 1Γ— B300 | Three-run medians; overall target unmet. [Same-card benchmark](https://github.com/InferMatrixLab/drama-3-single-b300-engine/blob/58a1882a4d941f8d21bfce6be37d3d5e7c9fde80/docs/current-results.md) | +| **h3-metal** | **βˆ’59%**
Baseline: h3.c Β· 73.3 s | Warm video generation latency Β· Apple M5 Max | Both engines used the original MiniMax-H3 at 20 steps. [Author benchmark](https://github.com/zuiho-kai/h3-metal/blob/53b7fa5d88e2497ab427ab9c7155535f7e6e6817/README.md#performance) | [Run your first experiment](#quickstart) Β· [Benchmark details and historical records](docs/optimization-results.md) Β· [vLLM-Omni model optimization roadmap](https://github.com/InferMatrixLab/InferRSI/issues/416) -## Documentation + + +## πŸ“š Documentation + +- **▢️ Run and review.** [Operations](docs/operations.md) Β· [Packaged optimization benchmarks](docs/optimization-benchmarks.md) Β· [Result delivery](docs/result-delivery.md) +- **🧠 Use knowledge.** [Skill library](src/inferrsi/knowledge/data/skill_library/README.md) Β· [CANNbot setup and revocation](docs/external-channels.md#setting-one-up) Β· [MindIE setup and revocation](docs/codex-plugin.md#enable-and-revoke) +- **🧭 Understand and extend.** [Module layout and import migration](docs/module-layout.md) Β· [Architecture and directory map](docs/architecture.md) Β· [Standalone engine template](docs/template/README.md) +- **πŸ“Œ Follow changes.** [Releases](https://github.com/InferMatrixLab/InferRSI/releases) Β· [Changelog](CHANGELOG.md) -- **Run and review.** [Operations](docs/operations.md) Β· [Packaged optimization benchmarks](docs/optimization-benchmarks.md) Β· [Result delivery](docs/result-delivery.md) -- **Use knowledge.** [Skill library](src/inferrsi/knowledge/data/skill_library/README.md) Β· [CANNbot setup and revocation](docs/external-channels.md#setting-one-up) Β· [MindIE setup and revocation](docs/codex-plugin.md#enable-and-revoke) -- **Understand and extend.** [Module layout and import migration](docs/module-layout.md) Β· [Architecture and directory map](docs/architecture.md) Β· [Standalone engine template](docs/template/README.md) -- **Follow changes.** [Releases](https://github.com/InferMatrixLab/InferRSI/releases) Β· [Changelog](CHANGELOG.md) + -## Contributing +## 🀝 Contributing -**People decide in issues; agents write the PRs.** +**🀝 People decide in issues; agents write the PRs.** People agree on the scope, resources, and acceptance criteria in the issue. Agents implement the agreed scope, run tests, and submit PRs with evidence. diff --git a/docs/optimization-results.md b/docs/optimization-results.md index 6ea8a360b..749c06a54 100644 --- a/docs/optimization-results.md +++ b/docs/optimization-results.md @@ -2,9 +2,10 @@ [Back to InferRSI](../README.md#selected-results) Β· [vLLM-Omni model optimization roadmap](https://github.com/InferMatrixLab/InferRSI/issues/416) -The tables show measured improvements of at least 20%. Hardware, workloads and -baseline revisions differ across comparisons. Quality and acceptance status are -reported for each result. +The README highlights measured improvements of at least 20%. This page keeps the +full curated record, including the campaigns below that bar. Hardware, workloads +and baseline revisions differ across comparisons. Quality and acceptance status +are reported for each result. ## Text serving @@ -17,7 +18,7 @@ limit. [Project report, configuration and reproduction](https://github.com/jiara | Scenario | Per-card throughput metric | Latency limit | Baseline β†’ optimized (tokens/s) | Measured change | | --- | --- | --- | --- | --- | -| 19 Β· Prefill | Input throughout the run | Mean TTFT < 10 s | 68,239 β†’ 105,397 | **+54.45%** | +| 19 Β· Prefill | Input throughput | Mean TTFT < 10 s | 68,239 β†’ 105,397 | **+54.45%** | | 21 Β· Decode, C1152 | Steady-state output | Mean TPOT < 50 ms | 1,711.98 β†’ 2,330.28 | **+36.12%** | | 23 Β· End-to-end, steady state | Decode output | Mean TPOT < 21 ms | 1,132.9 β†’ 1,368.5 | **+20.80%** | @@ -57,7 +58,7 @@ are reported directly by upstream contributors, with validation scoped to their | **MiniMax-H3-Image** | 2Γ— RTX 3090 |

  • Rank-local layerwise offload
  • Selected DiT weights kept resident
  • AdaLN caching
  • Request-local text refinement reuse
| **36.62 β†’ 14.59 s (βˆ’60.16%)**
Warm mean HTTP delivery latency | One upstream A/B of complete deployment profiles. All nine image pairs had identical decoded RGB pixels, including warmups. Repeated qualification remains incomplete. | [Paired deployment benchmark](https://github.com/vllm-project/vllm-omni/pull/10243#:~:text=One%20paired%20deployment%20comparison) | [#10243](https://github.com/vllm-project/vllm-omni/pull/10243) (open) | | **MiniCPM-o 4.5** | 1Γ— L20X | Seed-TTS runtime and deployment changes | **11.46 β†’ 22.99 requests/s (+100.6%)** and **0.65548 β†’ 0.32300 mean RTF (βˆ’50.72%)** at C32
**10.62 β†’ 18.51 requests/s (+74.29%)** and **0.34287 β†’ 0.19744 mean RTF (βˆ’42.42%)** at C16
**2.94 β†’ 3.66 requests/s (+24.49%)** at C1 | Five retained fresh servers per arm, 12,800 successful requests. The independent-engine target was outside this comparison. Formal speech-quality non-inferiority remains unestablished. | [Campaign #431](https://github.com/InferMatrixLab/InferRSI/issues/431) | [#10331](https://github.com/vllm-project/vllm-omni/pull/10331) (merged) | | **MiniCPM-o 4.5** | 1Γ— L20X |
  • Packed conditioning on a tuned configuration
  • TensorRT speaker extraction + CUDA MPS (separate comparison)
| **βˆ’27.69%** median TTFA
Packed conditioning, C32, same 50-frame window
**+20.5%** at C16 Β· **+25.4%** at C32
TensorRT + MPS throughput | The packed-conditioning candidate missed the exploratory WER margin against the tuning reference at the 50-frame setting. The separate TensorRT + MPS comparison had speech-quality verdict `FAIL`. | [Campaign #49](https://github.com/InferMatrixLab/InferRSI/issues/49) | [#9218](https://github.com/vllm-project/vllm-omni/pull/9218), [#9188](https://github.com/vllm-project/vllm-omni/pull/9188) (both merged) | -| **LFM3-Omni-2B** | 1Γ— Ascend 910B1 |
  • Depth graphs
  • Grouped greedy sampling
| **0.285 β†’ 0.593 requests/s (+108.4%)**
Upstream offline Omni API A/B, C4, temperature 0 | Retained audio waveforms matched bit-for-bit within each tested temperature. | [Offline A/B benchmark](https://github.com/vllm-project/vllm-omni/pull/9827#:~:text=Performance%20results) | [#9827](https://github.com/vllm-project/vllm-omni/pull/9827) (merged) | +| **LFM3-Omni-2B** | 1Γ— Ascend 910B1 |
  • Depth graphs
  • Grouped greedy sampling
| **0.285 β†’ 0.593 requests/s (+108.1%)**
Upstream offline Omni API A/B, C4, temperature 0 | Retained audio waveforms matched bit-for-bit within each tested temperature. | [Offline A/B benchmark](https://github.com/vllm-project/vllm-omni/pull/9827#:~:text=Performance%20results) | [#9827](https://github.com/vllm-project/vllm-omni/pull/9827) (merged) | | **Drama 3** | 2Γ— Ascend 910B1 | Full graph replay for steady-state decode | **17.31 β†’ 22.40 / 22.91 frames/s (+31%)** and **1.237 β†’ 0.962 / 0.947 mean RTF (βˆ’23%)** at C1 | Rounded changes use the mean of the two candidate runs. The real-time factor targets were unmet. | [Campaign #132](https://github.com/InferMatrixLab/InferRSI/issues/132) | [#9308](https://github.com/vllm-project/vllm-omni/pull/9308) (merged) | | **Qwen3-TTS 1.7B** | 1Γ— Ascend 910B1 | Codec and predictor optimizations | **1.869 β†’ 4.822 audio s/s (+158%)** at C1
**16.463 β†’ 38.755 (+135%)** at C16
**21.245 β†’ 52.091 (+145%)** at C32 | Historical seven-path profile vs. fresh same-card no-PR control. Mandarin speaker similarity and some ASR results regressed. Upstream reference. | [vLLM-Omni roadmap #416](https://github.com/InferMatrixLab/InferRSI/issues/416) | [#9068](https://github.com/vllm-project/vllm-omni/pull/9068) (merged) | @@ -71,6 +72,17 @@ import, so acceptance remains `INCONCLUSIVE`. The 30% target was unmet. The B300 campaign and this cohort used different hardware and measurement protocols. [Earlier Ascend report](qwen-image-2.1-turbo-ascend.md) +### Historical campaigns below the 20% highlight bar + +These campaigns stay on the record even though their measured gain is under the +bar the README highlights. Their upstream PR labels are kept as they were. + +| Model | Hardware | Measured change | Validation and limits | Evidence | vLLM-Omni PR | +| --- | --- | --- | --- | --- | --- | +| **MiniMax-H3 Ref2VA** | 4 logical Ascend 910C devices | **291.25 β†’ 286.67 s (βˆ’1.57%)**
5 s video, HTTP end-to-end | One lossless screening pair; strict quality checks passed for the screened candidate. Campaign `INCONCLUSIVE`. | [Campaign #297](https://github.com/InferMatrixLab/InferRSI/issues/297) Β· [Evidence PR #317](https://github.com/InferMatrixLab/InferRSI/pull/317) | [#9745](https://github.com/vllm-project/vllm-omni/pull/9745) (draft) | +| **Qwen3-Omni** | 2Γ— L20X | **+7.79%** throughput Β· **βˆ’6.06%** mean RTF
One C64 pilot | Median time to first audio regressed 7.12%. Campaign `INCONCLUSIVE`. | [Campaign #112](https://github.com/InferMatrixLab/InferRSI/issues/112) Β· [Pinned pilot](https://github.com/InferMatrixLab/InferRSI/tree/e1ae256fc075ea9ab64aec68da749cf4a8c22ac7/evidence/qwen3-omni-2gpu-20261023/results/pr9283-cudnn-performance) | [#9283](https://github.com/vllm-project/vllm-omni/pull/9283) (draft) | +| **LingBot-World** | 4Γ— Ascend 910C1 | **108.29 β†’ 103.89 s (βˆ’4.06%)**
Full-request latency, upstream AB/BA | Pre-codec tensors and decoded RGB frames matched exactly. | [KV gather benchmark](https://github.com/vllm-project/vllm-omni/pull/8112#:~:text=Performance%3A%20whole%2Dblock%20index_select%20KV%20gather%20on%20NPU) | [#8112](https://github.com/vllm-project/vllm-omni/pull/8112) (merged) | + ### Historical MiniMax-H3 Ref2VA step-reduction measurements on 950PR The author reported these 5-second workloads on four Ascend 950PR cards @@ -89,7 +101,9 @@ whole workload change. [Record #520](https://github.com/InferMatrixLab/InferRSI/ | 11-step dense | 4.74Γ— | **βˆ’78.9%** | | 12-step dense | 4.34Γ— | **βˆ’77.0%** | -TTFA is time to first audio. RTF is real-time factor. WER is word error rate. +TTFA is time to first audio. RTF is real-time factor. WER is word error rate. The +LFM3-Omni-2B change is computed from the rounded means above; upstream quotes ++108.4% from its unrounded values. Upstream PR statuses checked on 2026-11-20. To refresh a row, substitute its PR number in `gh pr view 10184 --repo vllm-project/vllm-omni --json state,isDraft`. @@ -109,7 +123,7 @@ Author-reported results on **Apple M5 Max (40-core GPU), 128 GB, macOS 27**, at | **h3-metal**, original H3 | Apple M5 Max (40-core GPU) | 20 | **54.29 s** | **29.89 s** | **2.82Γ— / 4.50Γ—** | **β€” / 2.45Γ—** | | **h3-metal**, FastH3 4-step + block-sparse attention | Apple M5 Max (40-core GPU) | 4 ΒΉ | **48.23 s** | **18.20 s** | **3.17Γ— / 7.40Γ—** | **1.40Γ— / 4.03Γ—** | -Cold includes the first clip's weight loading, text encoding and preparation; warm reuses loaded weights and prepared state. +Cold includes the first clip's weight loading, text encoding and preparation; warm reuses loaded weights and prepared state. h3-metal's cold time is 19.7% below h3.c's, just under the 20% bar this page highlights. ΒΉ FastH3 uses a distilled 4-step checkpoint, unlike the 20-step configurations. Warm values average clips 2–4 for the 20-step rows and two A/B/B/A warm runs for FastH3. From a777b007bbd4e4c9074c46f93c4091222c2e4c00 Mon Sep 17 00:00:00 2001 From: Dong1017 <55723162+Dong1017@users.noreply.github.com> Date: Fri, 20 Nov 2026 22:39:39 +0800 Subject: [PATCH 6/7] docs: keep one fire mark per table, move the LFM3 note to its row - README: the legend says the fire mark is the largest gain in its table, so only the 4.22x row, the +158% row and the +381% row keep it. - optimization-results: the sentence about the LFM3 percentage cited "the rounded means above", which sat under the 950PR step-reduction table. Move it into the LFM3 row itself. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Dong1017 <55723162+Dong1017@users.noreply.github.com> --- README.md | 6 +++--- docs/optimization-results.md | 6 ++---- 2 files changed, 5 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 8cc0afa8a..3a17468f1 100644 --- a/README.md +++ b/README.md @@ -116,9 +116,9 @@ listed baseline. The evidence column gives the scope and validation status. | Model | Measured change and baseline | Metric and hardware | Validation and evidence | | --- | --- | --- | --- | -| **MiniMax-H3 Ref2VA** | **βˆ’73.45%**
Baseline: Dense inference Β· 407.27 s | 5 s video, HTTP latency Β· 8Γ— Ascend 910B1 | **πŸ”₯ 3.77Γ—** with lossy Cache-DiT. βœ… 14 / 14 quality comparisons passed. [Campaign](https://github.com/InferMatrixLab/InferRSI/issues/293) Β· [Benchmark](evidence/h3-ref2va-ascend-293/results/final-5s/README.md) | +| **MiniMax-H3 Ref2VA** | **βˆ’73.45%**
Baseline: Dense inference Β· 407.27 s | 5 s video, HTTP latency Β· 8Γ— Ascend 910B1 | **3.77Γ—** with lossy Cache-DiT. βœ… 14 / 14 quality comparisons passed. [Campaign](https://github.com/InferMatrixLab/InferRSI/issues/293) Β· [Benchmark](evidence/h3-ref2va-ascend-293/results/final-5s/README.md) | | **MiniMax-H3 Ref2VA** | **βˆ’76.3%**
Baseline: In-run dense control Β· 701.95 s | 5 s video, 50-step denoising time Β· 4Γ— Ascend 950PR | **πŸ”₯ 4.22Γ—** in five retained A/B slots. ⚠️ Historical; performance and quality inconclusive. [Archive record](https://github.com/InferMatrixLab/InferRSI/issues/520) Β· [Recipe benchmark](https://github.com/vllm-project/vllm-omni/pull/9955#:~:text=in%2Drun%20dense%20median) | -| **MiniMax-H3-Image** | **βˆ’60.16%**
Baseline: Original layer offload Β· 36.62 s | Warm HTTP latency Β· 2Γ— RTX 3090 | **πŸ”₯** One paired comparison; βœ… 9 pixel-identical outputs. [Upstream results](https://github.com/vllm-project/vllm-omni/pull/10243#:~:text=One%20paired%20deployment%20comparison) | +| **MiniMax-H3-Image** | **βˆ’60.16%**
Baseline: Original layer offload Β· 36.62 s | Warm HTTP latency Β· 2Γ— RTX 3090 | One paired comparison; βœ… 9 pixel-identical outputs. [Upstream results](https://github.com/vllm-project/vllm-omni/pull/10243#:~:text=One%20paired%20deployment%20comparison) | | **MiniMax-H3-Image** | **βˆ’35.9% / βˆ’36.1%**
Baseline: Same-layout SP4 control Β· 16.89 / 4.18 s | HTTP generation latency at 4736Γ—3552 / 1024Γ—1024 Β· 4Γ— Ascend 910B1 | βœ… 30 / 30 numerical checks passed. Latency targets unmet. [Benchmark](https://github.com/InferMatrixLab/InferRSI/pull/394) | | **Qwen-Image-2.1-Turbo** | **βˆ’30.60%**
Baseline: Pinned control Β· 384.16 ms | Warm HTTP latency Β· 4Γ— B300 | βœ… 44 pixel-identical outputs. [InferRSI results](docs/qwen-image-2.1-turbo-b300.md) | | **Qwen-Image-2.1-Turbo** | **βˆ’22.20%**
Baseline: Upstream parent Β· 2,921.23 ms | Warm offline generation latency Β· 1Γ— Ascend 910B1 | ⚠️ Upstream reference; 18 pixel-identical outputs. Teardown inconclusive. [Upstream results](https://github.com/vllm-project/vllm-omni/pull/8729#:~:text=Final%2Dmerge%20performance%20and%20image%20comparison) | @@ -129,7 +129,7 @@ listed baseline. The evidence column gives the scope and validation status. | Model | Measured change and baseline | Metric and hardware | Validation and evidence | | --- | --- | --- | --- | -| **LFM3-Omni-2B** | **+108.1%**
Baseline: Functional eager adapter Β· 0.285 req/s | Offline greedy request throughput Β· 1Γ— Ascend 910B1, C4 | **πŸ”₯** βœ… Retained audio bit-identical at every tested temperature. [Upstream results](https://github.com/vllm-project/vllm-omni/pull/9827#:~:text=Performance%20results) | +| **LFM3-Omni-2B** | **+108.1%**
Baseline: Functional eager adapter Β· 0.285 req/s | Offline greedy request throughput Β· 1Γ— Ascend 910B1, C4 | βœ… Retained audio bit-identical at every tested temperature. [Upstream results](https://github.com/vllm-project/vllm-omni/pull/9827#:~:text=Performance%20results) | | **MiniCPM-o 4.5** | **+100.6% / βˆ’50.72%**
Baseline: Pinned upstream main Β· 11.46 req/s / 0.65548 RTF | Request throughput / mean real-time factor Β· 1Γ— L20X, C32 | ⚠️ Five matched runs per arm; speech-quality non-inferiority unestablished. [Campaign](https://github.com/InferMatrixLab/InferRSI/issues/431) Β· [Benchmark](https://github.com/vllm-project/vllm-omni/pull/10331) | | **MiniCPM-o 4.5** | **βˆ’27.69%**
Baseline: Tuned 50-frame deployment before packed conditioning | Median time to first audio Β· 1Γ— L20X, C32 | ⚠️ Missed the exploratory word-error-rate margin. [Benchmark](https://github.com/vllm-project/vllm-omni/pull/9218) | | **MiniCPM-o 4.5** | **+20.5% / +25.4%**
Baseline: Pinned upstream main before TensorRT + MPS | Throughput at C16 / C32 Β· 1Γ— L20X | ⚠️ Speech quality check failed. [Benchmark](https://github.com/vllm-project/vllm-omni/pull/9188) | diff --git a/docs/optimization-results.md b/docs/optimization-results.md index 749c06a54..0a30ce97d 100644 --- a/docs/optimization-results.md +++ b/docs/optimization-results.md @@ -58,7 +58,7 @@ are reported directly by upstream contributors, with validation scoped to their | **MiniMax-H3-Image** | 2Γ— RTX 3090 |
  • Rank-local layerwise offload
  • Selected DiT weights kept resident
  • AdaLN caching
  • Request-local text refinement reuse
| **36.62 β†’ 14.59 s (βˆ’60.16%)**
Warm mean HTTP delivery latency | One upstream A/B of complete deployment profiles. All nine image pairs had identical decoded RGB pixels, including warmups. Repeated qualification remains incomplete. | [Paired deployment benchmark](https://github.com/vllm-project/vllm-omni/pull/10243#:~:text=One%20paired%20deployment%20comparison) | [#10243](https://github.com/vllm-project/vllm-omni/pull/10243) (open) | | **MiniCPM-o 4.5** | 1Γ— L20X | Seed-TTS runtime and deployment changes | **11.46 β†’ 22.99 requests/s (+100.6%)** and **0.65548 β†’ 0.32300 mean RTF (βˆ’50.72%)** at C32
**10.62 β†’ 18.51 requests/s (+74.29%)** and **0.34287 β†’ 0.19744 mean RTF (βˆ’42.42%)** at C16
**2.94 β†’ 3.66 requests/s (+24.49%)** at C1 | Five retained fresh servers per arm, 12,800 successful requests. The independent-engine target was outside this comparison. Formal speech-quality non-inferiority remains unestablished. | [Campaign #431](https://github.com/InferMatrixLab/InferRSI/issues/431) | [#10331](https://github.com/vllm-project/vllm-omni/pull/10331) (merged) | | **MiniCPM-o 4.5** | 1Γ— L20X |
  • Packed conditioning on a tuned configuration
  • TensorRT speaker extraction + CUDA MPS (separate comparison)
| **βˆ’27.69%** median TTFA
Packed conditioning, C32, same 50-frame window
**+20.5%** at C16 Β· **+25.4%** at C32
TensorRT + MPS throughput | The packed-conditioning candidate missed the exploratory WER margin against the tuning reference at the 50-frame setting. The separate TensorRT + MPS comparison had speech-quality verdict `FAIL`. | [Campaign #49](https://github.com/InferMatrixLab/InferRSI/issues/49) | [#9218](https://github.com/vllm-project/vllm-omni/pull/9218), [#9188](https://github.com/vllm-project/vllm-omni/pull/9188) (both merged) | -| **LFM3-Omni-2B** | 1Γ— Ascend 910B1 |
  • Depth graphs
  • Grouped greedy sampling
| **0.285 β†’ 0.593 requests/s (+108.1%)**
Upstream offline Omni API A/B, C4, temperature 0 | Retained audio waveforms matched bit-for-bit within each tested temperature. | [Offline A/B benchmark](https://github.com/vllm-project/vllm-omni/pull/9827#:~:text=Performance%20results) | [#9827](https://github.com/vllm-project/vllm-omni/pull/9827) (merged) | +| **LFM3-Omni-2B** | 1Γ— Ascend 910B1 |
  • Depth graphs
  • Grouped greedy sampling
| **0.285 β†’ 0.593 requests/s (+108.1%)**
Computed from the rounded means; upstream quotes +108.4% from its unrounded values.
Upstream offline Omni API A/B, C4, temperature 0 | Retained audio waveforms matched bit-for-bit within each tested temperature. | [Offline A/B benchmark](https://github.com/vllm-project/vllm-omni/pull/9827#:~:text=Performance%20results) | [#9827](https://github.com/vllm-project/vllm-omni/pull/9827) (merged) | | **Drama 3** | 2Γ— Ascend 910B1 | Full graph replay for steady-state decode | **17.31 β†’ 22.40 / 22.91 frames/s (+31%)** and **1.237 β†’ 0.962 / 0.947 mean RTF (βˆ’23%)** at C1 | Rounded changes use the mean of the two candidate runs. The real-time factor targets were unmet. | [Campaign #132](https://github.com/InferMatrixLab/InferRSI/issues/132) | [#9308](https://github.com/vllm-project/vllm-omni/pull/9308) (merged) | | **Qwen3-TTS 1.7B** | 1Γ— Ascend 910B1 | Codec and predictor optimizations | **1.869 β†’ 4.822 audio s/s (+158%)** at C1
**16.463 β†’ 38.755 (+135%)** at C16
**21.245 β†’ 52.091 (+145%)** at C32 | Historical seven-path profile vs. fresh same-card no-PR control. Mandarin speaker similarity and some ASR results regressed. Upstream reference. | [vLLM-Omni roadmap #416](https://github.com/InferMatrixLab/InferRSI/issues/416) | [#9068](https://github.com/vllm-project/vllm-omni/pull/9068) (merged) | @@ -101,9 +101,7 @@ whole workload change. [Record #520](https://github.com/InferMatrixLab/InferRSI/ | 11-step dense | 4.74Γ— | **βˆ’78.9%** | | 12-step dense | 4.34Γ— | **βˆ’77.0%** | -TTFA is time to first audio. RTF is real-time factor. WER is word error rate. The -LFM3-Omni-2B change is computed from the rounded means above; upstream quotes -+108.4% from its unrounded values. +TTFA is time to first audio. RTF is real-time factor. WER is word error rate. Upstream PR statuses checked on 2026-11-20. To refresh a row, substitute its PR number in `gh pr view 10184 --repo vllm-project/vllm-omni --json state,isDraft`. From 13c1d4a37c684fa5c4b26f25442715228c6a3822 Mon Sep 17 00:00:00 2001 From: Dong1017 <55723162+Dong1017@users.noreply.github.com> Date: Sat, 21 Nov 2026 00:00:57 +0800 Subject: [PATCH 7/7] docs: scope the DeepSeek throughput row to throughput and latency --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 3a17468f1..877c255c9 100644 --- a/README.md +++ b/README.md @@ -108,7 +108,7 @@ listed baseline. The evidence column gives the scope and validation status. | Model | Measured change and baseline | Metric and hardware | Validation and evidence | | --- | --- | --- | --- | -| **DeepSeek V4.1 Flash** | **+54.45% / +36.12% / +20.80%**
Baseline: 68,239 / 1,711.98 / 1,132.9 tokens/s per card | Per-card throughput, prefill / decode / end-to-end Β· Ascend A3, W4A8, 1P1D | βœ… Rechecked at each scenario's latency limit; decode rechecked once. [Workloads and evidence](https://github.com/jiaran-king/dsv4.1-optimize) | +| **DeepSeek V4.1 Flash** | **+54.45% / +36.12% / +20.80%**
Baseline: 68,239 / 1,711.98 / 1,132.9 tokens/s per card | Per-card throughput, prefill / decode / end-to-end Β· Ascend A3, W4A8, 1P1D | Throughput and latency rechecked at each scenario's latency limit; decode rechecked once. Long-output accuracy not verified. [Workloads and evidence](https://github.com/jiaran-king/dsv4.1-optimize) |