InferRSI helps coding agents and engineers profile inference workloads, test optimizations against a baseline, and verify output quality before adopting a change. Bring an engine checkout, a workload, and your quality checks.
🖥️ Use the CLI. Install from source and list the packaged benchmarks on a development machine with Python 3.10+.
git clone https://github.com/InferMatrixLab/InferRSI.git
cd InferRSI
python -m venv .venv
source .venv/bin/activate
python -m pip install .
inferrsi benchmarks listFrom a target vLLM-Omni checkout, run the CPU readiness probe.
inferrsi doctor --repo . --repo-type vllm_omni --backend cpuSetup and task discovery run without starting a benchmark. For a target host,
select its backend (cpu, cuda, rocm, ascend, or mlx).
Install guide · Release wheels and checksums
🤖 Use a coding agent. Install the plugin and ask it to check the target checkout, workload, and quality criteria before starting an experiment. Codex · DeepSeek Harness · WorkSwarm
🧪 Run your first experiment. Try a packaged benchmark · Bring your own workload
Outer loop. To improve the engine, the agent profiles the workload and proposes a change. InferRSI checks quality and measures the candidate against the baseline under the same frozen protocol. Round evidence guides the next change. A fresh process retests the best candidate at the end of the campaign.
Inner loop. To improve the optimizer, compare paired campaigns with knowledge on and off to measure whether access to the library helps the agent. Reviewed PRs add findings to the library for agents to consult in later campaigns. Knowledge comparison
- Knowledge supplies source-linked skills, prior measurements, and implementation notes for the agent to consult.
- Harness freezes the experiment inputs and gates, runs the rounds, and records the verdict and evidence.
- Benchmark supplies packaged tasks, measurements, and acceptance rules. Project campaigns supply their own workload and checks.
Apps assembles these capabilities into the CLI, client integrations, and result views. Module responsibilities · Workflow diagram · Architecture decision #362 · Apps migration #371
Review a run in the dashboard. Compare baseline and candidate profiles and
trace each experiment back to its evidence. From a recorded run directory,
open it with inferrsi dashboard --input ., which writes dashboard.html beside the run.
Dashboard guide · Download the interactive example
Dashboard demo with synthetic data.
Model bring-up and measured optimizations across text, image, video, and speech.
Performance tables highlight metrics that improved by at least 20% against the
listed baseline. The evidence column gives the scope and validation status.
🔥 marks the largest gain in its table, ✅ a row whose quality checks passed, and
| Model | Engine and hardware | Milestone | Evidence |
|---|---|---|---|
| DeepSeek V4.1 Flash | vLLM-Ascend · 8× Ascend 910C | W8A8 service on the 8K-input / 1K-output workload. Six independent runs gave 39.28 / 44.07 / 56.86 ms/token at C1 / C4 / C16; the 25 ms/token target remains unmet. | Bring-up and baseline |
| Qwen3.8-27B | Standalone nano-vLLM-derived engine · 1× H20 | FP8 service passed scoped HTTP/SSE lifecycle checks; quality and performance qualification incomplete. | Service bring-up · Performance campaign |
| LFM3-Omni-2B | vLLM-Omni · 1× Ascend 910B1 | ✅ Two real-weight serving tests passed; a functional eager adapter served as the offline audio baseline. | Serving checks and A/B benchmark |
| Model | Measured change and baseline | Metric and hardware | Validation and evidence |
|---|---|---|---|
| DeepSeek V4.1 Flash | +54.45% / +36.12% / +20.80% Baseline: 68,239 / 1,711.98 / 1,132.9 tokens/s per card |
Per-card throughput, prefill / decode / end-to-end · Ascend A3, W4A8, 1P1D | Throughput and latency rechecked at each scenario's latency limit; decode rechecked once. Long-output accuracy not verified. Workloads and evidence |
| Model | Measured change and baseline | Metric and hardware | Validation and evidence |
|---|---|---|---|
| MiniMax-H3 Ref2VA | −73.45% Baseline: Dense inference · 407.27 s |
5 s video, HTTP latency · 8× Ascend 910B1 | 3.77× with lossy Cache-DiT. ✅ 14 / 14 quality comparisons passed. Campaign · Benchmark |
| MiniMax-H3 Ref2VA | −76.3% Baseline: In-run dense control · 701.95 s |
5 s video, 50-step denoising time · 4× Ascend 950PR | 🔥 4.22× in five retained A/B slots. |
| MiniMax-H3-Image | −60.16% Baseline: Original layer offload · 36.62 s |
Warm HTTP latency · 2× RTX 3090 | One paired comparison; ✅ 9 pixel-identical outputs. Upstream results |
| MiniMax-H3-Image | −35.9% / −36.1% Baseline: Same-layout SP4 control · 16.89 / 4.18 s |
HTTP generation latency at 4736×3552 / 1024×1024 · 4× Ascend 910B1 | ✅ 30 / 30 numerical checks passed. Latency targets unmet. Benchmark |
| Qwen-Image-2.1-Turbo | −30.60% Baseline: Pinned control · 384.16 ms |
Warm HTTP latency · 4× B300 | ✅ 44 pixel-identical outputs. InferRSI results |
| Qwen-Image-2.1-Turbo | −22.20% Baseline: Upstream parent · 2,921.23 ms |
Warm offline generation latency · 1× Ascend 910B1 |
| Model | Measured change and baseline | Metric and hardware | Validation and evidence |
|---|---|---|---|
| LFM3-Omni-2B | +108.1% Baseline: Functional eager adapter · 0.285 req/s |
Offline greedy request throughput · 1× Ascend 910B1, C4 | ✅ Retained audio bit-identical at every tested temperature. Upstream results |
| MiniCPM-o 4.5 | +100.6% / −50.72% Baseline: Pinned upstream main · 11.46 req/s / 0.65548 RTF |
Request throughput / mean real-time factor · 1× L20X, C32 | |
| MiniCPM-o 4.5 | −27.69% Baseline: Tuned 50-frame deployment before packed conditioning |
Median time to first audio · 1× L20X, C32 | |
| MiniCPM-o 4.5 | +20.5% / +25.4% Baseline: Pinned upstream main before TensorRT + MPS |
Throughput at C16 / C32 · 1× L20X | |
| Qwen3-TTS 1.7B | +158% Baseline: Fresh no-PR control · 1.869 audio s/s |
Streaming audio throughput · 1× Ascend 910B1, C1 | 🔥 |
| Drama 3 | +31% / −23% Baseline: Piecewise graphs · 17.31 frames/s / 1.237 RTF |
Throughput / mean real-time factor · 2× Ascend 910B1, C1 | Two runs per arm; real-time-factor targets unmet. Campaign · Benchmark |
Engine projects report these comparisons. Speech benchmarks use one concurrent request per engine and measure audio frames per second and median time to first audio.
| Engine | Measured change and baseline | Metric and hardware | Benchmark scope and evidence |
|---|---|---|---|
| Qwen3-TTS | +381% / −81% Baseline: vLLM-Omni · 12.71 frames/s / 263.51 ms |
Audio throughput / time to first audio · 1× Ascend 910B1 | 🔥 ✅ Three runs per engine; audio and contract checks passed. Same-card benchmark |
| MiniCPM-o 4.5 | +112% / −57% Baseline: vLLM-Omni · 187.52 frames/s / 295.3 ms |
Audio throughput / time to first audio · 1× L20X | |
| Drama 3 | +58% / −31% Baseline: vLLM-Omni · 195.89 frames/s / 47.29 ms |
Audio throughput / time to first audio · 1× B300 | Three-run medians; overall target unmet. Same-card benchmark |
| h3-metal | −59% Baseline: h3.c · 73.3 s |
Warm video generation latency · Apple M5 Max | Both engines used the original MiniMax-H3 at 20 steps. Author benchmark |
Run your first experiment · Benchmark details and historical records · vLLM-Omni model optimization roadmap
▶️ Run and review. Operations · Packaged optimization benchmarks · Result delivery- 🧠 Use knowledge. Skill library · CANNbot setup and revocation · MindIE setup and revocation
- 🧭 Understand and extend. Module layout and import migration · Architecture and directory map · Standalone engine template
- 📌 Follow changes. Releases · Changelog
🤝 People decide in issues; agents write the PRs.
People agree on the scope, resources, and acceptance criteria in the issue. Agents implement the agreed scope, run tests, and submit PRs with evidence. To merge, a PR must pass the required checks and receive two independent approvals. The issue owner uses the evidence to accept the work or reopen it.
Start with a reproducible workload, a baseline, and evidence of the bottleneck.
Sign off commits with git commit -s.
Open an issue · Human–agent collaboration · Contributor guide · License
Run the local development checks before submitting a change.
python -m pip install -e ".[dev]"
pytest
ruff check src tests