Robotic Video World Models: A Survey of Applications, Research Challenges, Future Directions

arXiv:2601.07823 · eess.SY, cs.RO, cs.SY · Submitted 2026-01-12 · Read on arXiv

eess.SY, cs.RO, cs.SY

Submitted: 2026-01-12

Updated: 2026-09-17

Code: https://github.com/irom-princeton/awesome-robotics-video-world-model-papers

Terminology

• World Model — 100× in this paper · explained in 7 episodes
• Consistency — 44× in this paper · explained in 3 episodes
• Fidelity — 38× in this paper · explained in 2 episodes
• Benchmark — 34× in this paper · explained in 2 episodes
• Diffusion Model — 30× in this paper · explained in 8 episodes
• Diffusion Models — 25× in this paper · explained in 6 episodes
• Hallucination — 23× in this paper · explained in 3 episodes
• Reinforcement Learning — 22× in this paper · explained in 4 episodes
• Generalization — 19× in this paper · explained in 13 episodes
• Latent Space — 16× in this paper · explained in 7 episodes
• Transformer — 15× in this paper · explained in 2 episodes
• Alignment — 14× in this paper · explained in 1 episode
• Temporal consistency — 9× in this paper · explained in 2 episodes
• Affordance — 8× in this paper · explained in 1 episode
• Embeddings — 8× in this paper · explained in 2 episodes
• Generative Model — 7× in this paper · explained in 2 episodes
• Generative Models — 7× in this paper · explained in 2 episodes
• Core — 6× in this paper · explained in 2 episodes
• Fine-Tuning — 6× in this paper · explained in 5 episodes
• Optical Flow — 6× in this paper · explained in 2 episodes
• Foundation Model — 5× in this paper · explained in 3 episodes
• Gaussian Splatting — 5× in this paper · explained in 4 episodes
• Loss Function — 5× in this paper · explained in 2 episodes
• Uncertainty Quantification — 5× in this paper · explained in 8 episodes
• Accuracy — 4× in this paper · explained in 1 episode
• Action Space — 4× in this paper · explained in 1 episode
• CVF International Conference on Computer Vision — 4× in this paper
• Foundation Models — 4× in this paper · explained in 7 episodes
• Sim-to-Real Gap — 4× in this paper · explained in 6 episodes
• Abstraction — 3× in this paper · explained in 1 episode
• Grounding — 3× in this paper · explained in 3 episodes
• Latent Representation — 3× in this paper · explained in 2 episodes
• Latent Representations — 3× in this paper · explained in 3 episodes
• Robustness — 3× in this paper · explained in 5 episodes
• Unified Framework — 3× in this paper · explained in 5 episodes
• 3D Gaussian Splatting — 2× in this paper · explained in 2 episodes
• Adaptation — 2× in this paper · explained in 1 episode
• Benchmarking — 2× in this paper · explained in 3 episodes
• Contrastive Learning — 2× in this paper · explained in 5 episodes
• Data Augmentation — 2× in this paper · explained in 7 episodes
• Deep Reinforcement Learning — 2× in this paper · explained in 2 episodes
• International Conference on Robotics and Automation — 2× in this paper
• Latent Diffusion Model — 2× in this paper · explained in 2 episodes
• Latent Diffusion Models — 2× in this paper · explained in 2 episodes
• Quantization — 2× in this paper · explained in 12 episodes
• Reinforcement Learning (RL) — 2× in this paper · explained in 25 episodes
• Self-Supervised Learning — 2× in this paper · explained in 7 episodes
• Steering — 2× in this paper · explained in 2 episodes
• Verification — 2× in this paper · explained in 2 episodes
• Zero-Shot Generalization — 2× in this paper · explained in 5 episodes
• alpha — 2× in this paper · explained in 1 episode
• Actor — 1× in this paper · explained in 3 episodes
• Attention Mechanism — 1× in this paper · explained in 2 episodes
• Catastrophic Forgetting — 1× in this paper · explained in 5 episodes
• Closed-Loop Control — 1× in this paper · explained in 2 episodes
• Computational overhead — 1× in this paper · explained in 2 episodes
• Cosine Similarity — 1× in this paper · explained in 3 episodes
• Cross-Embodiment Transfer — 1× in this paper · explained in 2 episodes
• Decoupling — 1× in this paper · explained in 2 episodes
• Diffusion Process — 1× in this paper · explained in 2 episodes
• Distillation — 1× in this paper · explained in 2 episodes
• Downstream Tasks — 1× in this paper · explained in 2 episodes
• Embodied AI — 1× in this paper · explained in 2 episodes
• Flow Matching — 1× in this paper · explained in 7 episodes
• Generative Adversarial Networks (GANs) — 1× in this paper · explained in 3 episodes
• Generative Modeling — 1× in this paper · explained in 3 episodes
• Hallucination Detection — 1× in this paper · explained in 2 episodes
• Hallucination Mitigation — 1× in this paper · explained in 2 episodes
• Hierarchical Framework — 1× in this paper · explained in 2 episodes
• Instruction Tuning — 1× in this paper · explained in 2 episodes
• Interpretability — 1× in this paper · explained in 15 episodes
• Kullback-Leibler (KL) Divergence — 1× in this paper · explained in 3 episodes
• Latent Dynamics — 1× in this paper · explained in 2 episodes
• Mean Maximum Rank Violation — 1× in this paper
• Mode Collapse — 1× in this paper · explained in 2 episodes
• Model Predictive Control (MPC) — 1× in this paper · explained in 5 episodes
• Nash Equilibrium — 1× in this paper · explained in 2 episodes
• Objective Function — 1× in this paper · explained in 2 episodes
• Point Clouds — 1× in this paper · explained in 2 episodes
• Prompt Engineering — 1× in this paper · explained in 4 episodes
• Regularization — 1× in this paper · explained in 3 episodes
• Representation Learning — 1× in this paper · explained in 2 episodes
• Reward Function — 1× in this paper · explained in 2 episodes
• Sample Efficiency — 1× in this paper · explained in 4 episodes
• Segmentation Masks — 1× in this paper · explained in 2 episodes
• Semantic Alignment — 1× in this paper · explained in 3 episodes
• Sparse Attention — 1× in this paper · explained in 5 episodes
• Standardized Benchmarks — 1× in this paper · explained in 2 episodes
• Structural Constraints — 1× in this paper · explained in 2 episodes
• System Identification — 1× in this paper · explained in 3 episodes
• Task Decomposition — 1× in this paper · explained in 2 episodes
• Task Planning — 1× in this paper · explained in 2 episodes
• Test-time Scaling — 1× in this paper · explained in 2 episodes
• Transformer Architecture — 1× in this paper · explained in 2 episodes
• VLA Models — 1× in this paper · explained in 3 episodes
• Variational Bayes — 1× in this paper · explained in 2 episodes
• Vision-Language Models (VLMs) — 1× in this paper · explained in 6 episodes

Sources

• Flow Matching for Generative Modeling
• DiffuEraser: A Diffusion Model for Video Inpainting
• Wan: Open and Advanced Large-Scale Video Generative Models
• Cosmos World Foundation Model Platform for Physical AI
• DreamGen: Unlocking Generalization in Robot Learning through Video World Models
• NovaFlow: Zero-Shot Manipulation via Actionable Flow from Generated Videos
• Video Language Planning
• FLIP: Flow-Centric Generative Planning as General-Purpose Manipulation World Model
• WorldGym: World Model as An Environment for Policy Evaluation
• Ctrl-World: A Controllable Generative World Model for Robot Manipulation
• $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization
• Gemini Robotics: Bringing AI into the Physical World
• Vid2World: Crafting Video Diffusion Models to Interactive World Models
• EnerVerse: Envisioning Embodied Future Space for Robotics Manipulation
• Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulation
• Evaluating Gemini Robotics Policies in a Veo World Simulator
• Training Agents Inside of Scalable World Models
• World-Env: Leveraging World Model as a Virtual Environment for VLA Post-Training
• Learning to Act from Actionless Videos through Dense Correspondences
• Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control

Related papers

• One Request, Multiple Experts: LLM Orchestrates Domain Specific Models via Adaptive Task Routing
• A Geometric Decision Procedure for STL Feasibility and Repair
• Submodular Multi-Agent Policy Learning for Online Distributed Task Allocation in Open Multi-Agent Systems
• Policy-Level Recursive Self-Improvement for Embodied AI with a Criticality World Model
• Minimal Experiments for Robust Stabilization: Information, Spectral Geometry, and Duration
• Decentralized Power-Optimal Coordination for Spacecraft Swarms Using Time-Varying Magnetorquer Actuation