CoPE: Co-Evolving Acting and Forecasting for Language Agents

Introduction

CoPE is a reinforcement-learning stack for LLM agents in interactive environments, built on AgentGym-RL's fork of verl.

Agentic RL benchmarks hand back a single reward at the end of a trajectory, which leaves most of a batch with no gradient. CoPE adds an action forecasting loss: at every step the agent predicts the next K actions it will take, and the realized rollout supervises that prediction. The forecasts are then distilled back into the policy, so acting and forecasting improve each other. Both are off by default — with the flags off, the training path is stock GRPO.

Five environments are supported: SciWorld, ALFWorld, WebShop, AppWorld and τ²-bench.

Get started

Clone with the submodule

The AgentGym submodule carries every environment server and the agentenv base classes, so it is required for any run.

git clone --recurse-submodules https://github.com/ulab-uiuc/CoPE.git
cd CoPE
# if you already cloned without it:
git submodule update --init AgentGym

Set up the environments

Training and the environment servers run as separate processes with separate interpreters — WebShop needs Python 3.8 and τ²-bench needs ≥3.12, neither of which the 3.10 trainer can host. Each environment is its own HTTP service that training reaches over requests.

# the trainer (Python 3.10 + vllm); see AgentGym-RL's install guide for the details
conda create -n agentgym-rl python=3.10

# one env per environment you intend to run, e.g. SciWorld
conda create -n agentenv-sciworld python=3.10

Configure

The launchers read everything from environment variables, so a variant is one extra variable rather than a forked script.

export CONDA_SH=$HOME/miniconda3/etc/profile.d/conda.sh
export TRAIN_ENV=agentgym-rl           # conda env for the trainer
export SCIWORLD_ENV=agentenv-sciworld  # conda env for the env server
export MODEL_PATH=Qwen/Qwen2.5-7B-Instruct
export CUDA_VISIBLE_DEVICES=0,1,2,3

API keys are read from files under .secrets/ (git-ignored), never passed on a command line — scontrol show job and any env dump in a log would otherwise expose them:

mkdir -p .secrets && echo "sk-..." > .secrets/openai_api_key

Run

The launch_*_grpo_tmux.sh scripts bring up the environment servers and training together, each in its own tmux session, and wait for the servers to become healthy before training starts.

# baseline GRPO
bash scripts/launch_sciworld_grpo_tmux.sh

# with action forecasting
ACTION_FORECAST_ENABLE=True ACTION_FORECAST_COEF=0.01 \
  bash scripts/launch_sciworld_grpo_tmux.sh

# temporal ensembling, which distils the forecasts back into the policy
TE_ENABLE=True TE_LAMBDA=0.1 TE_ETA=0.5 \
  bash scripts/launch_sciworld_grpo_tmux.sh

Two published baselines are configured as plain config rather than env vars:

bash examples/train/AgentGym-RL/sciworld_train.sh
bash examples/train/ScalingInter-RL/sciworld_train.sh

Evaluation configs live in examples/eval/. τ²-bench runs under the published protocol and has its own scoring and trajectory viewer — see docs/TAU2_GRPO.md, which also carries the measured results (negative so far, reported rather than omitted) and two traps worth knowing before comparing any numbers against a paper.

License

MIT, see LICENSE. The vendored code under src/verl/ carries its own Apache 2.0 headers from upstream verl.