HOPE is an open platform for humanoid robot table tennis, developed by Hitch Interactive (Intelligent Racing Inc.) in collaboration with the ROAR Platform at UC Berkeley. The challenge invites teams to deploy whole-body humanoid controllers that can rally a ping-pong ball against human opponents or other robots, using off-the-shelf humanoid hardware and an open-source perception and planning stack.
This repository contains the full HOPE stack for the Agibot A3: Isaac Lab training for a unified forehand/backhand whole-body policy, a no-spin planner (ROS 2), a MuJoCo evaluator with real ball physics, and an A3 deploy reference runner — plus the preserved HOPE reference design documents, the challenge rulebooks, and the Agibot-provided A3 starter materials under agibot/.
Start with this README, then follow QUICKSTART_A3_ISAAC.md. The rest of the repository is organized into four layers:
| Layer | What to read or run | Purpose |
|---|---|---|
| Required path | QUICKSTART_A3_ISAAC.md, hope_training/whole_body_tracking/scripts/prepare_a3_isaac_asset.py, agibot/URDF/A3T2.5-URDF-std-pingpang/, hope_training/whole_body_tracking/ |
Prepare the A3 Isaac asset, train the unified forehand/backhand policy (task=HOPEPingPong), export hope_pingpong.onnx, and evaluate success_rate. |
| Stable public contracts | A3_ASSETS.md, docs/interfaces/, docs/POLICY_INTERFACE.md, docs/PLANNER_INTERFACE.md |
Frame conventions, joint order, the 111-D observation / 31-D action policy IO, ROS topics, RacketCommand, and asset expectations that stay stable when you integrate your own code. |
| Deploy and simulation references | apps/a3_mujoco_serve/, a3_deploy/, agibot/ |
The self-contained deterministic MuJoCo → DLS IK → CSV → high-level A3 serve app, the public deploy contract and clean-room policy runner, and Agibot-provided A3 materials. |
| Background material | NatNet2ROS2/, VRPN2ROS2/, hope_ws/, mocap/, root HOPE_*_Reference_Setup.md, REFERENCE_DOCS.md, ROADMAP.md |
The independent raw-mocap adapters and HOPE ROS 2 planner workspace for arena integration, the mocap frame/topic docs, the preserved design documents, and current scope/direction. |
A fresh clone contains only tracked files. Generated Isaac assets, training logs,
checkpoints, exported .onnx files, and ROS build artifacts are git-ignored and
regenerated locally; Agibot-provided materials under agibot/ are tracked.
# 1. Prepare the A3 Isaac asset (bundled racket-equipped URDF, from the repo root)
python3 hope_training/whole_body_tracking/scripts/prepare_a3_isaac_asset.py --force
# 2. Train the unified forehand/backhand policy (inside your Isaac Lab Python environment)
cd hope_training/whole_body_tracking
source setup_train_env.sh # defines the `isaac_py` launcher for the Isaac Sim Python
python scripts/train.py task=HOPEPingPong algo=ppo headless=true
# 3. Export the deployable actor -> <run>/exported/hope_pingpong.onnx
python scripts/export_onnx.py --checkpoint logs/rsl_rl/hope_pingpong/<run>/model_<iter>.pt
# 4. Evaluate (real MuJoCo ball, continuous rally) -> {"success_rate": <float>}
python scripts/mujoco_eval_onnx.py --onnx logs/rsl_rl/hope_pingpong/<run>/exported/hope_pingpong.onnx
# 5. Watch the deploy reference runner drive the robot in MuJoCo
cp logs/rsl_rl/hope_pingpong/<run>/exported/hope_pingpong.onnx ../../a3_deploy/a3_deploy_example/models/
cd ../../a3_deploy/a3_deploy_example
bash scripts/run_pingpong_sim.shTo close the loop with the planner (ROS 2):
cd hope_ws && colcon build && source install/setup.bash
ros2 launch hope_bringup hope_bringup.launch.py use_fake_ball:=true # mocap-free smoke test
# in another terminal (same ROS env sourced):
cd a3_deploy/a3_deploy_example/reference
python -m a3_deploy_onnx_ref_pingpong --planner --view --realtimeHOPE provides two independently built ROS 2 motion-capture adapters:
NatNet2ROS2 for
OptiTrack Motive/NatNet and
VRPN2ROS2
for Chingmu CMTracker/MCServer. Both paths express every accepted ROS 2 header
timestamp in the adapter host computer's NTP-disciplined
RCL_SYSTEM_TIME/Unix epoch. Here “world clock” means absolute UTC/Unix wall
clock, not the ROS world coordinate frame.
| Adapter | Source timestamp | Conversion into the adapter host world clock | Trust requirement |
|---|---|---|---|
| NatNet2ROS2 | Motive CameraMidExposureTimestamp in the Motive QPC domain |
NatNet echo exchanges estimate the QPC-to-adapter-steady-clock mapping; the measured capture age is subtracted from the adapter's Chrony-disciplined RCL_SYSTEM_TIME. Motive's Windows wall clock is not used for this conversion. |
Mapping age and uncertainty must pass the configured gates. |
| VRPN2ROS2 | CMTracker/MCServer report timeval |
The source seconds/microseconds are decoded as Unix time and validated against the adapter's Chrony-disciplined RCL_SYSTEM_TIME; absolute-age and sliding minimum-age gates detect invalid or changed timing regimes. |
The VRPN server host must already share the same NTP epoch; VRPN itself does not synchronize clocks. |
The adapter computer and the humanoid robot computer must be synchronized to
the same approved NTP server or equivalent common time source before live
mocap control. Matching ROS message types or plausible-looking timestamps is
not evidence of synchronization. For the VRPN path, the CMTracker/MCServer host
must also be NTP-disciplined because its wall-clock timeval is carried on the
wire. NatNet's QPC mapping avoids depending on the Motive Windows wall clock,
but the adapter host and robot must still share UTC so the resulting ROS stamp
can be compared with robot state and future execution time.
The Agibot A3 implementation is under
agibot/ntp_sync/. Chrony disciplines the A3 HDU
system clock, and the supervised HDU-to-MDU PTP chain distributes that time to
the internal controller. See also the
full clock synchronization plan.
Internet/NTP loss does not block ordinary offline A3 applications, but the
robot is not qualified for external-mocap strike timing until both the adapter
and A3 clock-health gates pass. Clock drift or mixed domains can corrupt state
estimation, latency measurement, recording, calibration, and strike timing
even while ROS 2 topics appear healthy.
Before using a newly installed A3 pelvis marker shell, perform the one-time P1
calibration and save its constant 6-DOF P1 → pelvis_link transform to a
persistent JSON file. The mocap system supplies the dynamic world → P1
pose from the P1 rigid-body markers. Calibration additionally requires an
independent full-6DOF geometry_msgs/PoseStamped measurement of
world → pelvis_link, for example on /a3/calibration/pelvis_pose. The
calibrator synchronizes those two pose topics and estimates their constant
relative transform; it does not consume a Table topic or TF.
The checked-in A3 hardware bridge publishes /body_drive/pelvis_imu/data, but
an IMU does not provide absolute pelvis translation. It therefore is not a
producer for /a3/calibration/pelvis_pose. Before real-robot calibration, the
A3 integration must bridge an independent external 6-DOF tracker or state
estimator to that topic with header.frame_id: world and timestamps in the
same clock domain as /P1/pose. The MuJoCo-only
/sim/a3/pelvis_pose publisher may be used for simulation validation after
both inputs have been expressed in its odom frame; it is not a hardware
producer. Do not run p1_pelvis_tf_publisher while collecting calibration,
and never construct the reference pose from P1, because either would make
the measurement circular.
Collect while moving the pelvis smoothly through at least 0.10 m translation
and 10 degrees rotation for at least one second. Residual RMS alone only tests
repeatability; the calibrator separately gates measured motion excitation,
timestamp uniqueness, and pair skew so a stationary capture cannot pass.
OptiTrack uses NatNet echo-mapped camera_utc acquisition timestamps in the
adapter host's NTP-disciplined ROS epoch; both calibration sources must use
that same acquisition-time epoch.
cd hope_ws && colcon build --packages-select hope_bringup && source install/setup.bash
ros2 run hope_bringup p1_pelvis_calibrator \
--p1-topic /P1/pose \
--pelvis-topic /a3/calibration/pelvis_pose \
--reference-frame world \
--pelvis-frame pelvis_link \
--p1-frame P1 \
--output calibration/p1_to_pelvis.jsonAt normal runtime, load that JSON to publish the constant P1 → pelvis_link TF:
ros2 run hope_bringup p1_pelvis_tf_publisher \
--calibration-file calibration/p1_to_pelvis.jsonThe resulting chain is world → P1 → pelvis_link; the first transform remains
the live mocap measurement, while only the second is static. An optional
alternative is to absorb the saved correction into Motive's P1 pivot
definition. If that is done, disable the static publisher and verify by
rerunning calibration that the correction is approximately identity. Never
apply both corrections. The Table asset may be enabled separately for arena
setup when needed; this tool does not use it, and competition must not stream it.
See mocap/README.md
and docs/OPTITRACK.md.
The bundled motion clips under hope_training/motions/preprocessed/ are
reference-only placeholders — replace them with real forehand/backhand clips
(docs/REPLACE_MOTIONS.md) before training a policy you
intend to deploy. See QUICKSTART_A3_ISAAC.md for the full
install → train → export → evaluate → run loop, and
docs/RUN_ON_AGIBOT.md for the deploy path.
| Document | Description | Version |
|---|---|---|
| Motion Capture System Reference Setup | OptiTrack/NatNet and Chingmu/VRPN ROS 2 arena configuration, coordinate frames, competition rigid bodies (Ball, P1, P2; Table for calibration only), robot root-frame registration, and streaming pipelines |
v0.7 |
| 7DOF Racket Model-based Planner Reference Setup | Ball state estimation, trajectory prediction, and racket target planning (Stages 1–3 of the HITTER framework), reimplemented in the HOPE canonical frame | v0.1 |
| WBC Simulation Training Reference Setup | SMPL-X motion acquisition, GMR retargeting, BeyondMimic RL training pipeline for whole-body control (Stage 4), with dual-backend support for Isaac Lab and mjlab | v0.5 |
| Hardware Deployment Reference Setup | Platform-specific real-robot deployment paths (including legged_control2 and AimRT): ONNX inference, ROS 2 node graph, PD gain tuning, safety procedures, and competition workflow |
v0.2 |
The three documents at the repository root each contain a Section 0 prologue
listing implementation differences from the original HITTER work (see
References); the mocap pair documents its differences inline. Where a preserved
document disagrees with the shipped code, the code and the docs/ contracts win —
REFERENCE_DOCS.md is the index.
The competition rulebooks ship at the repository root: HOPE_AI_Challenge_2026_Rules_EN.docx (English) and HOPE_AI_Challenge_2026_Rules_ZH.docx (中文).
| Path | Purpose |
|---|---|
| README.md | Repository orientation and the shortest train → export → evaluate → run commands. New teams should start here. |
| QUICKSTART_A3_ISAAC.md | Step-by-step fresh-clone A3/Isaac setup. |
docs/ |
Task guides (TRAIN_POLICY, REPLACE_MOTIONS, RUN_ON_AGIBOT, OPTITRACK, EXTENDING_HOPE_PINGPONG, REMOVED_FROM_STARTER) and the public contracts (POLICY_INTERFACE, PLANNER_INTERFACE, docs/interfaces/). |
| A3_ASSETS.md | Asset map for the racket-equipped A3 URDF, the generated Isaac copy, joint order, and the Agibot-provided A3 reference materials. |
| REFERENCE_DOCS.md | Index of preserved architecture, rules, mocap, training, and deployment reference documents. |
| ROADMAP.md | What is shipped, what is out of scope by design, and what comes next. |
HOPE_*_Reference_Setup.md |
Preserved system design documents (planner / WBC training / hardware deployment). |
HOPE_AI_Challenge_2026_Rules_EN.docx, ..._ZH.docx |
Challenge rulebooks (English / 中文). |
configs/ |
The shared no-spin ball model (ball_physics.yaml) used by training, planner, and eval. |
hope_training/ |
The Isaac Lab training extension (whole_body_tracking/ with task cfg, train/export/eval scripts), placeholder motion clips (motions/preprocessed/), the canonical A3 joint order (config/joint_order_agibot_a3.yaml), and the ball-physics fitting tools (ball_physics_fit/). |
NatNet2ROS2/ |
Independent ROS 2 workspace for the OptiTrack/Motive NatNet adapter, named-pose interfaces, acquisition-time mapping, and driver tests. Build and launch it separately from hope_ws. |
VRPN2ROS2/ |
Independent ROS 2 workspace for the ChingMu/VRPN client, strict server-time/NTP validation, and raw per-tracker PoseStamped topics. |
hope_ws/ |
ROS 2 workspace: hope_planner (no-spin planner), hope_bringup (HOPE-side pose_to_posearray / optitrack_mct_relay adapters and fake-ball publishers), and hope_msgs (RacketCommand.msg). Raw acquisition lives in the two sibling adapter workspaces. |
a3_deploy/ |
Public deploy contract and clean-room reference runner (a3_deploy_example/), the MuJoCo/AimRT simulation fork (A3_MuJoCo_Sim/), and the optional user-supplied URDF override location (URDF/). |
apps/a3_mujoco_serve/ |
Self-contained deterministic serve contribution: official A3 MuJoCo model/racket contact, legal-serve physics search, DLS IK, all-joint CSV export, replay validation, and the PR #18 high-level A3 runtime. |
agibot/ |
Agibot-provided A3 bundle: the racket-equipped source URDF (URDF/A3T2.5-URDF-std-pingpang/), the vendor deploy example (code_deployment/), the MuJoCo/AimRT simulation reference (A3_MuJoCo_Sim/), and mounting hardware models (pku/). |
mocap/ |
Motion-capture frame/topic contract (mocap/README.md) and the preserved mocap reference documents (EN/ZH). |
┌──────────────────────────────┐ ┌──────────────────────────────┐
│ OptiTrack Motive │ │ Chingmu CMTracker / MCServer │
│ NatNet UDP │ │ VRPN server │
│ Ball / P1 / P2 │ │ Ball / P1 / P2 │
└──────────────┬───────────────┘ └──────────────┬───────────────┘
│ NatNet │ VRPN
▼ ▼
┌──────────────────────────────┐ ┌──────────────────────────────┐
│ NatNet2ROS2 (standalone) │ │ VRPN2ROS2 (standalone) │
│ /optitrack/poses │ │ /vrpn_mocap/<name>/ │
│ (NamedPoseArray) │ │ pose_id_<N> (PoseStamped) │
└──────────────┬───────────────┘ └──────────────┬───────────────┘
│ optitrack_mct_relay │ pose_to_posearray
└──────────────────┬────────────────┘
▼
┌─────────────────────────────┐
│ /poses: full rigid-body pose │
│ xyz + quaternion (xyzw) │
└──────────┬──────────────────┘
│
▼
┌─────────────────────────────┐
│ hope_planner (Stages 1–3) │
│ │
│ Ball state estimation │
│ → no-spin trajectory │
│ prediction │
│ → racket target planning │
│ (side, position, velocity,│
│ time_to_strike) │
└──────────┬──────────────────┘
│ /racket/command
│ (hope_msgs/RacketCommand)
▼
┌─────────────────────────────┐
│ Policy runner (Stage 4) │
│ │
│ hope_pingpong.onnx @ 50 Hz │
│ 111-D obs → 31-D action │
│ │
│ Receives: │
│ • RacketCommand │
│ • robot root pose │
│ • Joint encoders │
│ │
│ Outputs: │
│ • Robot joint │
│ position cmds │
└──────────┬──────────────────┘
│
▼
┌─────────────────────────────┐
│ Agibot A3 humanoid │
│ │
│ PD controller │
│ → joint torques │
└─────────────────────────────┘
During competition the motion-capture stream carries the named rigid bodies Ball, P1, and
P2 (Ball is always first in /poses; the default VRPN bringup aggregates only the ball); a
Table asset is used for calibration only and appears only in training-data recordings. The ball pose includes position (x, y, z) and quaternion orientation
(qx, qy, qz, qw); pitch/yaw/roll are derived display values. The shipped planner currently
uses only the ball position, while the full pose is preserved for validation and future
spin-aware estimation.
The diagram shows both supported source paths: Chingmu/VRPN uses the independent
VRPN2ROS2client +pose_to_posearray(the defaultmocap_backend:=vrpn), while OptiTrack/Motive uses NatNet through the independently builtNatNet2ROS2driver plus thehope_wsoptitrack_mct_relay(mocap_backend:=optitrack). Both feed the identical/posescontract, so everything below that hop is unchanged. See docs/OPTITRACK.md.
The same policy runner drives either the shipped MuJoCo simulation
(a3_deploy/a3_deploy_example/scripts/run_pingpong_sim.sh) or, via your own
licensed Agibot vendor deploy package, the real robot
(docs/RUN_ON_AGIBOT.md).
Racket tracking is prohibited. During competition the motion capture system streams the named rigid bodies Ball, P1, and P2 — the ball and the two humanoid marker-cluster frames. A calibrated static transform maps each P1/P2 frame to that robot's declared URDF root frame (pelvis on Unitree G1; pelvis_link on Agibot A3). The table is a calibrated static world origin (a Table asset is used during setup only and appears only in training-data recordings). The ball stream contains position (x, y, z) and orientation, represented in ROS 2 as a quaternion (qx, qy, qz, qw). Pitch, yaw, and roll are derived views, not fields carried by geometry_msgs/Pose. No reflective markers may be placed on the racket, the robot's hand, or the wrist link. Each robot must infer its paddle's 6-DOF pose through forward kinematics from its declared root frame plus joint encoders. This is a deliberate competition constraint that tests autonomous paddle control through the robot's internal body model.
Implementation scope. The preserved reference documents include robot-specific integration examples; the code currently shipped in this repository implements the Agibot A3 (31 actuated DOF) path end to end: Isaac Lab training of one unified forehand/backhand policy (HOPE-PingPong-AgibotA3-v0), MuJoCo evaluation with real ball physics, and a clean-room deploy reference runner, alongside Agibot's own deploy example and MuJoCo/AimRT simulation reference.
Open-source training stack. The WBC training pipeline is built entirely on open-source code: BeyondMimic (MIT license), from which the hope_training/whole_body_tracking/ extension derives, GMR (MIT license) for SMPL-X to robot retargeting, and GVHMR for monocular video-to-SMPL-X extraction. The HITTER paper's trained weights are not released; all training starts from scratch, and the bundled motion clips are placeholders to be replaced with your own retargeted swings (docs/REPLACE_MOTIONS.md).
The table below reports current repository implementation coverage.
| Robot | DOF | Status |
|---|---|---|
| Agibot Expedition A3 | 31 actuated | Shipped end-to-end path: Isaac Lab training, ONNX export, MuJoCo evaluation, deploy reference runner, plus the Agibot vendor deploy example. |
| Unitree G1 | 29 | Discussed in the preserved reference design documents (HOPE_*_Reference_Setup.md) only; no shipped code path. |
All contracts share a common world frame (ROS 2 REP 103):
- Origin: Near-side left corner of the table surface
- X: Toward opponent (along the 2.74 m table length)
- Y: Left (along the 1.525 m table width)
- Z: Up
- Table surface height: 0.76 m above floor
The motion-capture system must stream Z-up poses to match this convention (in OptiTrack Motive: Up Axis → Z). See docs/interfaces/frames.md and mocap/README.md.
Each piece has its own dependencies — install only what the step you are on needs:
- Training / export: NVIDIA Isaac Sim + Isaac Lab (with
rsl_rl), Python 3.10, PyTorch, CUDA GPU - MuJoCo evaluation / reference runner:
mujoco,onnxruntime,numpy(no GPU needed) - Planner workspace: ROS 2 Jazzy (
rclpy),numpy,pyyaml - Real arena: OptiTrack Motive streams NatNet through the independently built/launched
NatNet2ROS2workspace and the HOPE-sideoptitrack_mct_relay(mocap_backend:=optitrack; set Motive Up Axis = Z and see docs/OPTITRACK.md). Chingmu CMTracker/MCServer streams VRPN through the independentVRPN2ROS2workspace and HOPE-sidepose_to_posearray(mocap_backend:=vrpn). In either case configure the named 6-DOF competition rigid bodiesBall,P1, andP2;Tableis calibration-only and is not streamed during competition.
| Repository | Purpose |
|---|---|
| HybridRobotics/whole_body_tracking | BeyondMimic training code — upstream of hope_training/whole_body_tracking/ |
| YanjieZe/GMR | General Motion Retargeting (SMPL-X → robot) |
| zju3dv/GVHMR | Video-to-SMPL-X pose estimation |
| AimRT/aimrt | Agibot's lightweight robotics middleware, used by the A3 MuJoCo sim and vendor deploy example |
| google-deepmind/mujoco | MuJoCo — physics for the evaluator and the reference runner |
The preserved reference design documents cite additional G1-era repositories
(legged_control2, motion_tracking_controller, mjlab); see
REFERENCE_DOCS.md.
- Su, Z., Zhang, B., Rahmanian, N., Gao, Y., Liao, Q., Regan, C., Sreenath, K., & Sastry, S. S. (2025). HITTER: A HumanoId Table TEnnis Robot via Hierarchical Planning and Learning. arXiv:2508.21043v2. Project page
- SMASH: Mastering Scalable Whole-Body Skills for Humanoid Ping-Pong with Egocentric Vision (University of Hong Kong). arXiv:2604.01158. Paper
- Hu, M., Chen, W., Li, W., Mandali, F., He, Z., Zhang, R., Krisna, P., Christian, K., Benaharon, L., Ma, D., et al. (2025). PACE: Physics Augmentation for Coordinated End-to-end Reinforcement Learning toward Versatile Humanoid Table Tennis (Purdue TRACE Lab, ICRA 2026). arXiv:2509.21690. Code
- Liao, Q., et al. (2025). BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusion. arXiv:2508.08241v4. Project page
- Araújo, J. P., Ze, Y., Xu, P., Wu, J., & Liu, C. K. (2025). Retargeting Matters: General Motion Retargeting for Humanoid Motion Tracking. arXiv:2510.02252.
- Ze, Y., et al. (2025). LATENT: Learning Athletic Humanoid Tennis Skills from Imperfect Human Motion Data. arXiv:2603.12686.
- mjlab: A Lightweight Framework for GPU-Accelerated Robot Learning. arXiv:2601.22074.
- Peng, X. B., et al. (2024). SMPLOlympics: Sports Environments for Physically Simulated Humanoids. arXiv:2407.00187.
The HOPE open platform is developed with the support of our technical sponsors, whose humanoid and motion-capture hardware make the reference design possible:
- AgiBot (Zhiyuan Robotics) — humanoid robot platforms (Expedition A3 and related hardware). agibot.com
- ChingMu (青瞳视觉) — CHINGMU optical motion-capture systems (CMTracker / CMAvatar). chingmu.com
- OptiTrack — Leyard (NaturalPoint, Inc., a Leyard company) — OptiTrack optical motion-capture cameras and Motive software. optitrack.com
This project is licensed under the Apache License, Version 2.0. See LICENSE for the full terms.
Some starter materials are derived from or interoperate with third-party software and robot assets. See THIRD_PARTY_NOTICES.md for provenance and per-directory license notes.
Allen Yang, Co-founder and CTO, Hitch Interactive (Intelligent Racing Inc.); Chair, AI Racing ROAR Platform, UC Berkeley; Founding Executive Director, VIVE AR Center, UC Berkeley
Development team: Franco Huang (lead), Jeremy Wei, Yikang Yu, and Jiayi Zhu.