VINE: Taming Generative Control Policies for Reinforcement Learning

Paper  |  Website

Overview

VINE (Value-gradient Iterative Noise Exploration) enables stable end-to-end value-gradient fine-tuning of flow-matching policies. Instead of following a single denoising trajectory, VINE reconstructs a fresh interpolation state at every denoising step, creating a stable differentiable path for value-gradient backpropagation while preserving the original flow-matching semantics.

1. Installation

The validated setup is Linux, Python 3.12, CUDA 12, JAX 0.5.3, and OGBench 1.2.1. The experiments require an NVIDIA GPU; CPU installation is intended for tests and code inspection only. From the repository root:

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

The pinned requirements install the CPU JAX wheel. On a machine with a system CUDA 12 installation, enable the GPU backend with:

python -m pip install "jax[cuda12-local]==0.5.3"
python -c "import jax; print(jax.devices())"

Set MUJOCO_GL=egl for headless evaluation. W&B is optional: the launchers use offline logging by default, and WANDB_MODE=disabled turns it off completely.

2. Download the datasets

Put OGBench data under ./ogbench/data (override with OGBENCH_DATASET_DIR):

# Eight single-file OGBench domains (incl. cube-double)
bash scripts/download_datasets.sh standard

# Or only cube-double for the quick-start launcher
bash scripts/download_datasets.sh cube-double

# 100M shard datasets (~39 GB each)
bash scripts/download_datasets.sh cube100m
bash scripts/download_datasets.sh puzzle100m

Run bash scripts/download_datasets.sh --help for all targets. Large shard layouts are also documented here.

3. Train and reproduce

Every experiment uses job_submission/experiment_template.sh. Preset defaults supply the paper bc_alpha for single-file datasets; the two 100M benchmarks override the preset default of 1 to 1000.

RUN_GROUP=VINE_CUBE2 OGBENCH_ENV_KEY=cube_double_agnt \
  bash job_submission/experiment_template.sh

RUN_GROUP=VINE_CUBE100M OGBENCH_ENV_KEY=cube_quadruple_100m_agnt \
  OGBENCH_DATASET_DIR=./ogbench/data/cube-quadruple-play-100m-v0 BC_ALPHA_OVERRIDE=1000 \
  bash job_submission/experiment_template.sh

RUN_GROUP=VINE_ANTMAZE_LARGE OGBENCH_ENV_KEY=antmaze_large_agnt \
  bash job_submission/experiment_template.sh

RUN_GROUP=VINE_ANTMAZE_GIANT OGBENCH_ENV_KEY=antmaze_giant_agnt \
  bash job_submission/experiment_template.sh

RUN_GROUP=VINE_HUMANOID_MED OGBENCH_ENV_KEY=humanoid_medium_agnt \
  bash job_submission/experiment_template.sh

RUN_GROUP=VINE_HUMANOID_LGE OGBENCH_ENV_KEY=humanoid_large_agnt \
  bash job_submission/experiment_template.sh

RUN_GROUP=VINE_SCENE OGBENCH_ENV_KEY=scene_play_agnt \
  bash job_submission/experiment_template.sh

RUN_GROUP=VINE_PUZZLE33 OGBENCH_ENV_KEY=puzzle_3x3_agnt \
  bash job_submission/experiment_template.sh

RUN_GROUP=VINE_CUBE3 OGBENCH_ENV_KEY=cube_triple_agnt \
  bash job_submission/experiment_template.sh

RUN_GROUP=VINE_PZ44_100M OGBENCH_ENV_KEY=puzzle_4x4_100m_agnt \
  OGBENCH_DATASET_DIR=./ogbench/data/puzzle-4x4-play-100m-v0 BC_ALPHA_OVERRIDE=1000 \
  bash job_submission/experiment_template.sh

Acknowledgments

This codebase is built on top of FQL.

BibTeX

@article{yang2026vine,
  author = {Rushuai Yang and Zhuo Han and Houlin Li and Hecheng Wang and Zhichao Wu and Rui Zhang and Zhaowei Zhang and Zihong Chen and Xiaohan Yan and Chiming Liu and Yi Chen and Wei Shan and Maoqing Yao},
  title  = {VINE: Taming Generative Control Policies for Reinforcement Learning},
  conference = {arXiv Pre-print},
  year = {2026},
  url = {http://arxiv.org/abs/2607.10369},
}

License

Apache License 2.0. See LICENSE. Third-party license notices are provided in THIRD_PARTY_LICENSES.md.