Motion-Omni

Training and inference code for Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialogue.

A spoken dialogue model that natively emits facial expression together with hand, upper-body and lower-body motion. Motion is generated from the hidden states that produce the speech, so a response costs one forward pass rather than a speech pass followed by a separate motion model over the rendered audio.

• Paper: https://arxiv.org/abs/2609.04250
• Project page: https://step-out.github.io/Motion-Omni-Page/
• Data: https://huggingface.co/datasets/ChengqianMa/Motion-Omni

Status

This release covers the model definition, the four-stage training recipe, and single-turn inference — enough to read what the model actually does and to retrain it. Evaluation harnesses and released weights are still being prepared.

Layout

motion_omni/
  model/
    language_model/llava_her_qwen.py   # the model: LLM + speech + motion heads
    speech_encoder/                    # Whisper encoder (frozen)
    speech_projector/                  # encoder -> LLM embedding space
    speech_generator/                  # LLM hidden states -> speech units
    motion_generator/                  # speech hidden states -> motion codes
    flow/                              # speech units -> waveform (vocoder)
  datasets/dataset.py                  # per-task processors and collation
  train/                               # training entry point and trainer
  inference.py                         # load model, run one turn, decode motion
  render/, render_smplx_pbr.py         # SMPL-X rendering for qualitative output
scripts/
  train/stage1.sh .. stage4_joint.sh   # the four-stage recipe
  inference/run_single_turn.py         # batch runner over a manifest
  zero2.json, zero3.json               # DeepSpeed configs

Install

conda create -n motionomni python=3.10 -y && conda activate motionomni
pip install -r requirements.txt

Four assets are external and must be obtained separately:

Asset	Used for	Where
Qwen2.5-7B-Instruct	LLM backbone	HuggingFace
whisper-large-v3	speech encoder	HuggingFace
GLM-4-Voice decoder (flow.pt, hift.pt)	speech units to waveform	GLM-4-Voice release
Language of Motion (lom)	motion VQ-VAE and SMPL-X utilities	https://github.com/Motion-Language/language-of-motion

lom must be importable (pip install -e its repo, or put it on PYTHONPATH). The motion generator imports it lazily, so the package loads without it, but decoding motion codes will fail.

Point the runtime at the assets with environment variables:

export MOTIONOMNI_FLOW_CKPT=/path/to/flow.pt
export MOTIONOMNI_HIFT_CKPT=/path/to/hift.pt
export MOTIONOMNI_VOICE_CONFIG=/path/to/cosyvoice/vocab_16K.yaml
export MOTIONOMNI_LOM_VQ_CKPT=/path/to/vq_emage_speaker_2.ckpt
export MOTIONOMNI_SMPLX_DIR=/path/to/smplx_models   # rendering only

Training

Four stages, each starting from the previous checkpoint. Every path in the scripts is an environment variable with a placeholder default, so set them before launching:

export BACKBONE=/path/to/Qwen2.5-7B-Instruct
export WHISPER=/path/to/whisper-large-v3
export DATA_ROOT=/path/to/data
export CKPT_ROOT=/path/to/checkpoints

bash scripts/train/stage1.sh        # ASR: train the speech projector only
bash scripts/train/stage2.sh        # TTS: add the speech generator
bash scripts/train/stage3.sh        # TTSM: add the motion generator
bash scripts/train/stage4_joint.sh  # joint mixture over asr/tts/t2t/ttsm/s2sm

Stage 4 is where the paper's claim about joint training is exercised: the LLM, Speech Generator and Motion Generator are co-adapted under both objectives. Its learning rates match the released checkpoint:

Group	LR
base model	1e-7
speech projector	1e-6
speech generator	5e-7
motion generator	2e-6

The speech encoder stays frozen throughout.

Inference

scripts/inference/run_single_turn.py runs one turn per manifest row and writes, per sample, the response text, speech units, waveform, motion codes and decoded motion, plus timing:

PYTHONPATH=. python scripts/inference/run_single_turn.py \
  --checkpoint /path/to/checkpoint \
  --manifest /path/to/manifest.json \
  --output-dir /path/to/output

The timings it records are offline component times over a full utterance, not first-token or first-audio streaming latency; the field names say so.

To call the model directly:

from motion_omni.inference import load_motionomni, build_input_ids, decode_motion

model, tokenizer, audio_decoder = load_motionomni(checkpoint_path, device="cuda")
output_ids, speech_units, motion_pred = model.generate(input_ids, ...)

generate returns text token ids, the predicted speech units, and the motion prediction.

Data format

Each training sample carries a task field that selects a processor in motion_omni/datasets/dataset.py:

Task	Input	Supervises
asr	speech	transcript
tts	text	speech units
t2t	text	text
ttsm	text	speech units + motion codes
s2sm	speech	speech units + motion codes

Motion is stored as VQ codes over four body parts (face, hands, upper, lower), decoded to SMPL-X body parameters with FLAME expression coefficients.

Citation

@article{ma2026motionomni,
  title  = {Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialogue},
  author = {Ma, Chengqian and Tao, Wei and Zhang, Haoyu and Guo, Yiwen},
  year   = {2026},
  eprint = {2609.04250},
  archivePrefix = {arXiv}
}

License

To be announced with the full release. The training data derives from an existing speech-instruction corpus and from motion produced by a third-party teacher model, so the terms have to stay compatible with those upstream licences.