WCM: A World Critic Model for Vision-Language-Action Reinforcement Learning
Senyu Fei, Xiaopeng Yu, Siyin Wang, Xianzhong Zhao, Jingjing Gong, Xipeng Qiu
cs.RO, cs.CL, cs.CV
Submitted: 2026-07-31
Code: https://github.com/sylvestf/WCM
Project page: https://sylvestf.github.io/wcm-homepage
License: http://creativecommons.org/licenses/by/4.0/
The gist: Reinforcement learning (RL) post-training of Vision-Language-Action (VLA) models has shown strong promise for robotic manipulation.
Terminology
Abstract
Reinforcement learning (RL) post-training of Vision-Language-Action (VLA) models has shown strong promise for robotic manipulation. Among RL methods, critic-based approaches rely on a value estimator that predominantly operates on single-frame observations or single-frame VLM backbone latents, which is a fundamental mismatch with the partially observable nature of robot control. A naive approach to incorporate observation history into the critic incurs exponential complexity with high-dimensional visual space, and still fails because pure scalar-return regression provides insufficient supervision for learning cross-temporal dynamics. We identify the root cause as a state approximation problem: without an explicit world modeling objective, the critic's representation cannot capture the temporal structure needed for accurate value estimation. To address this, we propose the World Critic Model (WCM), built on a lightweight LeJEPA architecture; WCM jointly predicts future latent state and estimates values, such that the critic's representation is explicitly trained to capture temporal dynamics rather than merely regress scalar returns. WCM integrates seamlessly into both on-policy and off-policy training pipelines and is compatible with state-of-the-art VLA backbones including Pi0, Pi0.5, and OpenVLA-OFT. Extensive experiments on 149 tasks across four benchmarks demonstrate that WCM consistently achieves state-of-the-art performance in both in-distribution and out-of-distribution settings, with particularly strong generalization gains. We further validate WCM on seven real-world manipulation tasks using OpenVLA-OFT and Pi0.5 with off-policy RL, confirming stable deployment across diverse settings.
Sources
- $\pi_0$: A Vision-Language-Action Flow Model for General Robot Control
- ${\pi}_{0.7}$: a Steerable Generalist Robotic Foundation Model with Emergent Capabilities
- Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success
- OpenVLA: An Open-Source Vision-Language-Action Model
- What Can RL Bring to VLA Generalization? An Empirical Study
- SRPO: Self-Referential Policy Optimization for Vision-Language-Action Models
- SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning
- RLinf-VLA: A Unified and Efficient Framework for Reinforcement Learning of Vision-Language-Action Models
- $\pi^{*}_{0.6}$: a VLA That Learns From Experience
- $\pi_\texttt{RL}$: Online RL Fine-tuning for Flow-based Vision-Language-Action Models
- RLinf-USER: A Unified and Extensible System for Real-World Online Policy Learning in Embodied AI
- Predictive State Representations: A New Theory for Modeling Dynamical Systems
- VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning
- GPT-4 Technical Report
- Gemini: A Family of Highly Capable Multimodal Models
- PaLM-E: An Embodied Multimodal Language Model
- FAST: Efficient Action Tokenization for Vision-Language-Action Models
- Beyond Imitation: Reinforcement Learning-Based Sim-Real Co-Training for VLA Models
- Interactive Post-Training for Vision-Language-Action Models
- Proximal Policy Optimization Algorithms
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies
- Fine-tuning is Not Enough: A Parallel Framework for Collaborative Imitation and Reinforcement Learning in End-to-end Autonomous Driving