What Can Latent World Models Know? Physical Information in Multimodal Predictive Representations

arXiv:2607.27017 · cs.LG, cs.RO · Submitted 2026-07-29 · Read on arXiv

cs.LG, cs.RO

Submitted: 2026-07-29

Updated: 2026-09-26

Terminology

• Trace — 18× in this paper · explained in 2 episodes
• World Model — 17× in this paper · explained in 7 episodes
• AUC — 11× in this paper · explained in 2 episodes
• Latent World Models — 8× in this paper · explained in 2 episodes
• Linear probe — 6× in this paper · explained in 2 episodes
• Embeddings — 5× in this paper · explained in 2 episodes
• Regularization — 5× in this paper · explained in 3 episodes
• Degeneracy — 3× in this paper · explained in 3 episodes
• Transformer — 3× in this paper · explained in 2 episodes
• Benchmark — 2× in this paper · explained in 2 episodes
• Closed-Loop Control — 2× in this paper · explained in 2 episodes
• Parameter Identifiability — 2× in this paper · explained in 2 episodes
• Reinforcement Learning — 2× in this paper · explained in 4 episodes
• Self-Supervised Learning — 2× in this paper · explained in 7 episodes
• System Identification — 2× in this paper · explained in 3 episodes
• Alignment — 1× in this paper · explained in 1 episode
• Core — 1× in this paper · explained in 2 episodes
• Decode — 1× in this paper · explained in 2 episodes
• Foundation Model — 1× in this paper · explained in 3 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• Intrinsic Dimension — 1× in this paper · explained in 2 episodes
• Latent Space — 1× in this paper · explained in 7 episodes
• Representation Learning — 1× in this paper · explained in 2 episodes

Sources

• The SIGReg Objective as Variational Free Energy: A Theoretical Active-Inference Account of JEPA World Models
• Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture
• V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
• LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics
• A Generalization Theory for JEPA-Based World Models
• RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot
• Intuitive physics understanding emerges from self-supervised pretraining on natural videos
• Mastering Diverse Domains through World Models
• Sparsh: Self-supervised touch representations for vision-based tactile sensing
• LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels
• Do generative video models understand physical principles?
• Kepler-Encoder-v0.1: Towards a Multimodal Embedding Model for Robots
• Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformers
• TacGen: Touch Is a Necessary Dimension of Physical-World Representation -- Addressing Tactile Data Scarcity with Scalable Vision-to-Touch Alignment and Generation
• A Control Theory of Predictability in Latent World Models
• DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning
• TouchWorld: A Predictive and Reactive Tactile Foundation Model for Dexterous Manipulation

Related papers

• Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
• AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
• Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
• Convergence issues in Relational Concept Analysis based on AOC-posets
• Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
• Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks