ReactiveGWM: Steering NPC in Reactive Game World Models

A reactive demo on the SF3 game.

ReactiveGWM: Steering NPC in Reactive Game World Models
Zeqing Wang¹², Danze Chen¹², Zhaohu Xing⁴ , Zizhao Tong¹⁵ , Yinhan Zhang¹⁶ , Xingyi Yang³ , Yeying Jin^12
1 Tencent, ² National University of Singapore, ³ The Hong Kong Polytechnic University
⁴ The Hong Kong University of Science and Technology (Guangzhou), ⁵ University of Chinese Academy of Sciences, ⁶ The Hong Kong University of Science and Technology

⭐ Updates

• [Jun 16, 2026]: Causal Forcing training code is released, covering the ar_tf, cd, and dmd three-stage workflow.
• [Jun 11, 2026]: Bidirectional training code and documentation are released, covering bidirectional full DiT fine-tuning, LoRA fine-tuning, scoped module fine-tuning, and cached dataset precompute.
• [May 18, 2026]: Arxiv paper is released.
• [May 14, 2026]: Inference Code, model and dataset are released.

📚 Introduction

ReactiveGWM is a novel game world model that synthesizes dynamic interactions between the player and NPC. Unlike current player-centric game world models, ReactiveGWM explicitly decouples player control from NPC autonomy: player actions are injected into the diffusion backbone via a lightweight additive bias, while high-level NPC strategies (Offense, Control, Defense) are grounded through cross-attention modules. These modules learn a game-agnostic representation of interactive logic, enabling zero-shot strategy transfer to vanilla world models of different games without retraining. Experiments on Street Fighter 2 and Street Fighter Alpha 3 show ReactiveGWM delivers fine-grained player controllability alongside autonomous, prompt-aligned NPC behavior.

A reactive demo on the SF2 game.

🛠️ Setup

conda create -n ReactiveGWM python=3.10
conda activate ReactiveGWM
pip install -r requirements.txt

Download the base model, the ReactiveGWM checkpoints, and (optionally) the strategy-aligned datasets:

Resource	🤗 Repo
Wan2.2 base (VAE + T5)	Wan-AI/Wan2.2-TI2V-5B
UMT5 tokenizer	Wan-AI/Wan2.1-T2V-1.3B (google/umt5-xxl/ subfolder)
ReactiveGWM checkpoints (SF2 / SF3)	INV-WZQ/ReactiveGWM-Models
Strategy-aligned datasets	INV-WZQ/ReactiveGWM-Datasets

Arrange the base assets under a single <base_model_dir>:

<base_model_dir>/
└── Wan-AI/
    ├── Wan2.2-TI2V-5B/
    │   ├── Wan2.2_VAE.pth
    │   └── models_t5_umt5-xxl-enc-bf16.pth
    └── Wan2.1-T2V-1.3B/
        └── google/umt5-xxl/        # HF tokenizer files

🚀 Quick start

The repo ships 12 pre-built example samples under inference/examples/ (3 strategies × 2 samples × 2 variants). Pick any one and run:

SAMPLE=inference/examples/SF2/offense/01

python inference/inference.py \
    --variant sf2 --ckpt <path-to-sf2.safetensors> \
    --image     inference/examples/SF2/SF2.png \
    --actions   $SAMPLE/actions.parquet \
    --prompt    "$(cat $SAMPLE/prompt.txt)" \
    --base_model <base_model_dir> \
    --out out.mp4

The first frame is shared per variant (inference/examples/<variant>/<variant>.png); swap the --image along with --variant / --ckpt when moving from SF2 to SF3. Swap offense / control / defense to redirect NPC behavior with the same first frame and button stream.

For the full CLI argument table, the public Python API, module layout, and the per-strategy example inventory, see inference/README.md.

🏋️ Training

Training code is available under training/. It covers ordinary bidirectional training for SF2/SF3, including:

• full DiT fine-tuning;
• LoRA fine-tuning;
• scoped module fine-tuning via --trainable_filter / --trainable_filter_exclude;
• optional VAE/T5 cache precompute and cached-dataset training.

It also includes Causal Forcing training with the three-stage ar_tf, cd, and dmd workflow.

Training uses the local DiffSynth-Studio/diffsynth package as the underlying training framework. Install the inference requirements plus the training extras, then expose the project parent and DiffSynth-Studio on PYTHONPATH:

pip install -r requirements.txt
pip install -r training/requirements.txt
export PYTHONPATH=/path/to/ReactiveGWM/DiffSynth-Studio:/path/to/ReactiveGWM:${PYTHONPATH}

Validate the bidirectional entrypoints with:

python -m ReactiveGWM_Code.training.bidirectional.train --help
python -m ReactiveGWM_Code.training.bidirectional.precompute_cache --help
python -m ReactiveGWM_Code.training.causal_forcing.train --help

For full command examples, cache usage, and Causal Forcing stage commands, see training/README.md.

🤓 Acknowledgments

ReactiveGWM is built on top of Wan2.2-TI2V-5B and adapts modeling / scheduler components from DiffSynth-Studio. The Causal Forcing training workflow builds on ideas from thu-ml/Causal-Forcing. Gameplay recording uses the stable-retro framework, and NPC strategy annotations are produced by Gemini. We extend our gratitude to the open-source community for their valuable contributions!

🔗 Citation

@misc{wang2026reactivegwmsteeringnpcreactive,
      title={ReactiveGWM: Steering NPC in Reactive Game World Models}, 
      author={Zeqing Wang and Danze Chen and Zhaohu Xing and Zizhao Tong and Yinhan Zhang and Xingyi Yang and Yeying Jin},
      year={2026},
      eprint={2605.15256},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2605.15256}, 
}