Xianbao Hou^1,3,* · Zehao Ni^2,3,* · Yonghao He^3,† · Lingfeng Qian³
Xin Shi³ · Hu Su⁴ · Wei Sui^3,✉ · Cong Yang^1,✉

¹School of Future Science and Engineering, Soochow University
²Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University
³D-Robotics
⁴State Key Laboratory of Multimodal Artificial Intelligence Systems,
Institute of Automation, Chinese Academy of Sciences

^*Equal contribution   ^†Project lead   ^✉Corresponding authors

Wei Sui: wei.sui@d-robotics.cc
Cong Yang: cong.yang@suda.edu.cn

rtpolicy-demo-inline.mp4

Multi-view perception. Compact visual tokens. Four-step action generation.

RTPolicy turns multiple RGB views and robot state into action chunks with a compact flow-matching policy. It combines multi-view reasoning with a deployment path for edge devices, reaching 32.9 Hz on S600 in INT16 in the paper's latency evaluation.

🔥 Updates

RTPolicy code and demonstration are available.

🧭 1. Introduction

1.1 Brief Introduction of RTPolicy

RTPolicy fuses dense visual features across cameras before compressing them, retaining cross-view information in a small representation for the action model.

• Reason across views. A frozen DINOv3 ConvNeXt backbone and view-aware feature fusion combine complementary observations from three RGB cameras.
• Compress before action generation. In the paper's 480 × 640 (H × W) setting, visual compression reduces 900 visual tokens to 12, giving the DiT a compact observation representation.
• Generate action chunks in four steps. A flow-matching DiT conditions on the visual representation and robot state to predict actions with four integration steps.

1.2 Repository Structure

Path	What to find
rtpolicy/models/policy/rtpolicy/	Visual encoder, multi-view fusion, adapters, DiT, and policy runner
rtpolicy/datasets/	Dataset readers and preprocessing interfaces
rtpolicy/trainer/	Training and ONNX export
rtpolicy/deployment/	Backend adapters and asynchronous streaming with trajectory stitching
configs/	Training, observation, robot, and ONNX export configurations
simulation/robotwin2.0/	RoboTwin2.0 environments and RTPolicy adapter
scripts/ · tools/	Evaluation, inference, TensorRT tools, and dataset preparation
docs/	Runtime guide

📊 2. Model Overview

2.1 RoboTwin2.0 Simulation

The clean-to-clean benchmark uses 50 tasks, 50 clean demonstrations per task, 100 evaluation episodes per task, and task-specific checkpoints.

Method	Mean success rate · 50 tasks
DP	28.0%
ACT	29.7%
RDT	34.5%
π0	46.4%
RTPolicy	52.3%

Baseline scores are taken from the official single-task leaderboard.

Model checkpoints ↓

2.2 Latency on GPU

Platform	Precision	Latency	Frequency
RTX 4090	FP32	23.80 ms	42.0 Hz
RTX 4090	FP16	10.92 ms	91.6 Hz

2.3 Latency on S600 and S100

Platform	Precision	Latency	Frequency
S600	INT16	30.35 ms	32.9 Hz
S100	INT16	149.72 ms	6.7 Hz

Quantization HTML Reports

Device	Precision	File
S600	INT16
S100	INT16

2.4 Model Checkpoints

Model	Format	Google Drive
RoboTwin2 pretraining · 300K steps	PyTorch	Download
lift_pot · fine-tuned	PyTorch	Download
lift_pot · fine-tuned	ONNX FP32	Download
lift_pot · fine-tuned	TensorRT mixed FP16	Download

All checkpoint files. Each archive includes its configuration and usage notes.

The lift_pot evaluation configurations use NFE4, seed0. The TensorRT engine was built on NVIDIA L4 with TensorRT 11.1.0.106; its build environment is included in the archive.

🚀 3. Getting Started

3.1 Installation

git clone https://github.com/D-Robotics-AI-Lab/RTPolicy.git
cd RTPolicy
python -m pip install -r requirements.txt

The root requirements.txt covers PyTorch, RTPolicy, RoboTwin2, and ONNX. TensorRT can be added to the same CUDA 12 environment in Section 4.4.

3.2 Prepare Data and Weights

Supply compatible DINOv3 weights and set model.obs_encoder.emvis_config.backbone_path to their directory. Datasets, checkpoints, simulator assets, and generated outputs are kept outside Git.

Train directly on LeRobot v2.1 datasets with the default backend. Lance is optional; the Training Guide also covers converting LeRobot v3.0 data to JPEG-blob Lance.

For RoboTwin2.0 simulation, prepare SAPIEN, Vulkan, cuRobo, and the official assets:

cd simulation/robotwin2.0
bash script/_download_assets.sh
cd ../..

3.3 🤖 AI-assisted Setup and Validation

Give your coding agent the AI Setup Guide and an external artifact directory. It automatically downloads the released lift_pot checkpoint from Google Drive, verifies its checksum, installs the environment and validates an ONNX export. Paste this prompt after replacing the artifact path:

Follow https://github.com/D-Robotics-AI-Lab/RTPolicy/blob/main/docs/AI_SETUP.md for https://github.com/D-Robotics-AI-Lab/RTPolicy, using ARTIFACT_ROOT=/path/to/external/artifacts. Use an existing checkout if available; otherwise clone the repository.
Create an isolated compatible environment, install the root requirements.txt, and route downloads, checkpoints, caches and outputs outside Git.
Automatically download the released lift_pot PyTorch model from the Google Drive links in the guide, verify its pinned SHA256 and safely extract it. Reuse verified cached files. A local checkpoint directory is optional.
Inspect the checkpoint and export static FP32 ONNX with verification, using the released lift_pot protocol NFE4 and seed0 through configuration overrides; preserve checkpoint weights and stored config.json. If I provide another checkpoint or protocol, use those settings instead.
Validate the exported policy once and report action/action_pred errors against the same FP32 reference and identical inputs/noise. Label synthetic inputs explicitly.
If I request published ONNX or TensorRT validation, download and verify the matching package too. Check the TensorRT build environment before using its engine; rebuild from FP32 ONNX when the host requires it.
Run RoboTwin2 task evaluation only when I request it, through the external launcher with the simulator code unchanged. Validate each requested stage once.
Return reproducible commands, source/download identities and checksums, resolved versions/settings, real artifact paths and validation results. Report concrete blockers without substituting different models or claiming unrun tests.

⚡ 4. Training, Export and Inference

4.1 Train a Policy

Follow the Training Guide to train directly with LeRobot or use the optional Lance backend.

The RoboTwin2 pretraining and fine-tuning recipes use DINOv3 ConvNeXt-Base:

Stage	Configuration	Dataset	Training steps
Pretrain	Three-view Lance	50 tasks, 2500 demonstrations	300K
Fine-tune	Three-view LeRobot	50 demonstrations per task	30K

See Pretrain and fine-tune for configuration fields and commands.

4.2 Evaluate in RoboTwin2.0

Set the checkpoint and backend in the external evaluation configuration. The adapter uses RoboTwin2's original policy interface; simulator, task, and evaluation code remain unchanged.

python scripts/eval/run_robotwin2.py \
  --config configs/rtpolicy_eval/robotwin2.yaml \
  --runtime-dir /path/to/external/evaluation

JPEG input alignment is enabled by default through RTPOLICY_JPEG_ROUNDTRIP=1. Select backend, model_path, metadata_path, and test_num in the evaluation configuration. The launcher prepares an external working directory and calls the original evaluator. It does not alter preflight checks or seed selection. Keep checkpoint, NFE, seed, and task settings identical when comparing backends.

Build cuRobo v0.7.8 in an external directory using the same environment. Set WARP_CACHE_PATH outside the repository. If its filesystem rejects directory renames with EXDEV, run python tools/patch_warp_cache.py --apply in this isolated environment to enable an atomic file-copy fallback for the JIT cache.

git clone --branch v0.7.8 --depth 1 https://github.com/NVlabs/curobo.git /path/to/external/curobo
MAX_JOBS=1 python -m pip install -e /path/to/external/curobo \
  --no-build-isolation -r requirements.txt

4.3 Export and Run ONNX

Set checkpoint_dir and output_dir in the RoboTwin2 export example.

python main.py --config-path=configs/rtpolicy_onnx \
  --config-name=rtpolicy_onnx_robotwin2

python scripts/infer_onnx.py --help

The exported graph follows the checkpoint configuration, including its sampling settings. For a benchmark with an explicit NFE override, set num_inference_timesteps in the export YAML and use the same RTPOLICY_NFE during evaluation. To generate samples from a converted RoboTwin2 dataset, set train_dataset.root to its LeRobot v2.1 directory and choose num_calibration_samples. The example saves verification inputs and outputs under output_dir/onnx_calibration.

4.4 TensorRT and Edge Deployment

Use FP16 for NVIDIA TensorRT and INT16 for S600 / S100 BPU deployment.

Add the optional TensorRT tools to the installed environment:

python -m pip install -e '.[tensorrt]'

Set RTPOLICY_ARTIFACT_DIR to the export configuration's output_dir:

export RTPOLICY_ARTIFACT_DIR=/path/to/external/storage/rtpolicy

python scripts/fp16_convert.py \
  "$RTPOLICY_ARTIFACT_DIR/model.onnx" "$RTPOLICY_ARTIFACT_DIR/model-fp16.onnx"
python scripts/build_bench_x86.py build \
  "$RTPOLICY_ARTIFACT_DIR/model-fp16.onnx" "$RTPOLICY_ARTIFACT_DIR/model-fp16.engine" strongly_typed \
  --workspace-gib 1

python scripts/infer_tensorrt.py \
  --engine-path "$RTPOLICY_ARTIFACT_DIR/model-fp16.engine" \
  --inputs-dir "$RTPOLICY_ARTIFACT_DIR/onnx_calibration/sample_000/inputs" \
  --output-dir "$RTPOLICY_ARTIFACT_DIR/tensorrt_outputs"

python scripts/build_bench_x86.py bench \
  "$RTPOLICY_ARTIFACT_DIR/model-fp16.engine" \
  "$RTPOLICY_ARTIFACT_DIR/onnx_calibration/sample_000/inputs" 100

Conversion keeps normalization and other sensitive operators in FP32. The engine follows the ONNX graph's precision and requires static input shapes. The benchmark includes host transfers and allocation.

For RoboTwin2 inference, set these fields in the external evaluation YAML, then run the launcher in Section 4.2. Use the original FP32 export's metadata, whose public I/O is preserved by mixed-FP16 conversion:

backend: tensorrt
model_path: /path/to/artifacts/model-fp16.engine
metadata_path: /path/to/artifacts/model.onnx.metadata.json

4.5 Streaming Inference

Streaming runs prediction alongside action execution. The deployment design targets 10 Hz inference on the latest observation window, with a 30 Hz action-waypoint grid and piecewise-jerk optimization to join successive chunks.

Set ckpt_setting in the streaming example, then run:

python scripts/eval/run_robotwin2.py \
  --config configs/rtpolicy_eval/robotwin2_streaming.yaml \
  --runtime-dir /path/to/external/streaming-evaluation

The opt-in adapter reuses RoboTwin2's existing observation encoding, action execution and evaluator. Streaming guide explains the real-robot pipeline, trajectory stitching and simulation timebase.

A single 100-episode check reached 94/100 on lift_pot with PyTorch BF16, NFE4, seed0 and JPEG input alignment. The guide records the protocol and comparison.

🤝 Acknowledgements

We thank the authors of DINOv3 and VO-DP for sharing their research and code.

📄 License

See LICENSE for the repository terms. Third-party components retain their respective licenses.