Latent Video Prediction for World Modeling: An Evaluation Uncovering Intriguing Favorable Evidence

arXiv:2605.15618 · cs.CV, cs.AI · Submitted 2026-05-15 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "Latent Video Prediction for World Modeling".

Tom: This study provides a systematic evaluation of four matched-capacity frontier video foundation models—V-JEPA 2.1, V-JEPA 2, VideoPrism,

Jane: First, who's behind it and why it matters.

Title and authors: Tom: We're talking about the paper "Latent Video Prediction for World Modeling: An Evaluation Uncovering Intriguing Favorable Evidence" and how it systematically tests four frontier video foundation models—V-JEPA two point one, V-JEPA two VideoPrism, and VideoMAEv2—against these five crucial robustness axes. The main finding is that latent prediction models form a consistent profile across all these tests, suggesting they are inherently better suited for robust world modeling than other approaches we’ve seen before.

Jane: Exactly, Tom; the study highlights that these latent-prediction models degrade more gracefully when things get corrupted compared to other methods, and they manage to preserve usable class structures instead of just showing some superficial geometric stability under occlusion.

Lu: The paper points out that the joint-embedding predictive objective is what allows these latent models to discard surface-level visual variation in favor of higher-order semantic structure, which is a key mechanism at play here.

Meng: For practical application, it means when we deploy these systems in environments with imperfect sensors, like surveillance footage with some noise or partial views, we can expect their performance to hold up much better than models that just rely on raw pixel features.

Lalam: From a cultural standpoint, if we can develop world models that are robust to these kinds of real-world imperfections, it means the AI systems we build will be more reliable and less prone to catastrophic failure when interacting with the physical world.

The paper's summary: Tom: To summarize what the paper says in "Latent Video Prediction for World Modeling: An Evaluation Uncovering Intriguing Favorable Evidence," they are analyzing four matched-capacity frontier models—V-JEPA two point one, V-JEPA two VideoPrism, and VideoMAEv2—across feature discriminability, corruption robustness, fine-grained discrimination, occlusion robustness, and sensitivity to temporal direction. The main conclusion is that latent prediction models consistently show superior performance across all these five critical deployment axes.

Jane: It really boils down to this: the research isn't just looking at a single top-one score on clean data; it’s about understanding how different self-supervised pretraining objectives shape the internal representations of these video models.

Lu: Specifically, they establish that latent prediction models create a distinct and consistent profile because the joint-embedding predictive objectives capture higher-level semantic structure.

Meng: So, when we look at the results, it seems like the supervised TimeSformer baseline is competitive on clean metrics but then degrades significantly when you introduce corruption or occlusion, which proves that a single top-one score doesn't capture deployment-relevant weaknesses.

Lalam: This tells us that for building useful video world models, the way we train the model—whether it’s focusing on reconstruction or prediction—actually dictates how well it handles real-world chaos.

The paper's improvements: Tom: The authors suggest that a representation intended for use as a video world model needs to do more than just classify clean clips correctly; it has to remain reliable under sensor noise, tolerate missing input, distinguish actions that differ only in fine physical detail, and capture how a scene evolves over time, including the direction in which events unfold.

Jane: They are advocating for moving away from methods that might just focus on geometric stability or pixel reconstruction when dealing with these complex real-world demands.

Lu: The latent prediction objective is what allows the model to capture fine-grained contact cues without reconstructing pixels, which is something the paper shows V-JEPA does better than pixel reconstruction because it distills the spatiotemporal signatures that differentiate real from simulated contact.

Meng: That’s important for practical engineering; if a model can distinguish between "pushing" and "pulling" based on subtle physical contact cues instead of just looking at where the pixels are located, that adds a lot of fidelity to our interaction understanding.

Lalam: It really suggests that the future direction for world modeling research should be focused on objectives like joint-embedding latent prediction because they naturally encourage this kind of semantic understanding over just matching surface textures.

Conclusion: Tom: So, to wrap up the discussion on "Latent Video Prediction for World Modeling: An Evaluation Uncovering Intriguing Favorable Evidence," we see that latent prediction models consistently perform well across feature discriminability, corruption robustness, fine-grained discrimination, occlusion robustness, and temporal direction sensitivity.

Jane: It means these models degrade more gracefully under pixel corruption and can capture subtle physical cues without needing to reconstruct pixels or rely solely on geometric similarity when things are occluded.

Lu: The paper’s findings confirm that the joint-embedding predictive objective is what allows these latent prediction models to uniquely encode the arrow of time, creating an oriented temporal axis separating different actions like pushing and pulling by direction.

Meng: From an engineering standpoint, this means we can trust the decisions made by these latent models more when they are deployed in unpredictable settings where input quality is not guaranteed.

Lalam: It’s exciting because it moves us toward AI systems that possess an internal sense of physics and temporal causality, which is essential for reliable planning and understanding complex interactions.

Tom: Absolutely, the paper "Latent Video Prediction for World Modeling: An Evaluation Uncovering Intriguing Favorable Evidence" shows us a clear direction for how we should be evaluating these models beyond simple clean accuracy scores. That gives us a lot to think about as we look at what comes next in video AI research.

University of Melbourne · Monash University

cs.CV, cs.AI

Submitted: 2026-05-15

Updated: 2026-09-30

Comments: Accepted at NeurIPS 2026 (Evaluation and Dataset Track)

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: This study provides a systematic evaluation of four matched-capacity frontier video foundation models—V-JEPA 2.1, V-JEPA 2, VideoPrism, and VideoMAEv2—across five robustness axes relevant to

Key concepts

Feature Discriminability: This measures how well the model's internal representations can distinguish between different video concepts or actions. The study found that V-JEPA models create the most distinct features, meaning they capture high-level semantic structures better than other methods, which is crucial for accurate world understanding.
Corruption Robustness: This assesses how well the model maintains performance when the input video is intentionally corrupted (e.g., pixel noise or transforms). V-JEPA models are highly resilient because their training focuses on semantic structure rather than surface details, allowing them to retain accuracy even under heavy corruption.
Fine-Grained Action Discrimination: This tests the model's ability to detect subtle differences in complex actions, such as distinguishing between 'pretend' and 'real' physical contact. V-JEPA excels here because its latent prediction objective captures specific spatiotemporal cues that differentiate these nuanced interactions.
Temporal Directional Coherence: This refers to the model's ability to understand the direction of time in a video, such as distinguishing between 'pushing' and 'pulling'. V-JEPA models show superior directional encoding because their training forces them to learn an oriented temporal axis, which is vital for modeling dynamic interactions.

Terminology

Accuracy — 99× in this paper · explained in 1 episode
Robustness — 56× in this paper · explained in 5 episodes
Cosine Similarity — 26× in this paper · explained in 3 episodes
World Model — 26× in this paper · explained in 7 episodes
Corruption Robustness — 14× in this paper · explained in 2 episodes
Benchmark — 10× in this paper · explained in 2 episodes
Foundation Model — 9× in this paper · explained in 3 episodes
Foundation Models — 8× in this paper · explained in 7 episodes
Decoupling — 7× in this paper · explained in 2 episodes
Latent Space — 7× in this paper · explained in 7 episodes
Self-Supervised Learning — 6× in this paper · explained in 7 episodes
Latent Representation — 5× in this paper · explained in 2 episodes
Linear probe — 5× in this paper · explained in 2 episodes
Transformer — 5× in this paper · explained in 2 episodes
Consistency — 4× in this paper · explained in 3 episodes
Temporal Dependency Index — 4× in this paper
Temporal consistency — 4× in this paper · explained in 2 episodes
Adaptation — 3× in this paper · explained in 1 episode
Calibration — 3× in this paper · explained in 7 episodes
Decoupling Index — 3× in this paper
Embeddings — 3× in this paper · explained in 2 episodes
Fine-Tuning — 3× in this paper · explained in 5 episodes
Grounding — 3× in this paper · explained in 3 episodes
Latent Representations — 3× in this paper · explained in 3 episodes
Benchmarking — 2× in this paper · explained in 3 episodes
Contrastive Learning — 2× in this paper · explained in 5 episodes
Directional Semantic Coherence Score — 2× in this paper
Failure Modes — 2× in this paper · explained in 3 episodes
Hallucination — 2× in this paper · explained in 3 episodes
Representation Learning — 2× in this paper · explained in 2 episodes
AUC — 1× in this paper · explained in 2 episodes
Ablation — 1× in this paper · explained in 1 episode
Adversarial Robustness — 1× in this paper · explained in 3 episodes
Alignment — 1× in this paper · explained in 1 episode
Calibration Failure — 1× in this paper · explained in 2 episodes
Clustering — 1× in this paper · explained in 2 episodes
Distribution Shift — 1× in this paper · explained in 5 episodes
Fidelity — 1× in this paper · explained in 2 episodes
Interpretability — 1× in this paper · explained in 15 episodes
Joint-Embedding Predictive Architectures — 1× in this paper
Self-Supervised Learning (SSL) — 1× in this paper · explained in 4 episodes
Semantic Understanding — 1× in this paper · explained in 2 episodes

Summary

This study provides a systematic evaluation of four matched-capacity frontier video foundation models—V-JEPA 2.1, V-JEPA 2, VideoPrism, and VideoMAEv2—across five robustness axes relevant to their deployment as video world models: feature discriminability, corruption robustness, fine-grained discrimination, occlusion robustness, and sensitivity to temporal direction. The research addresses the gap in understanding these models by analyzing how different self-supervised pretraining objectives shape their representations rather than just measuring a single top-1 accuracy score on clean benchmarks. The findings establish that latent prediction models form a distinct and consistent profile, showing superior performance across these critical deployment axes, suggesting that latent prediction is more suitable for robust world modeling than pixel reconstruction or contrastive methods alone.

Evaluation Framework and Models

The study isolates the pretraining objective by holding three factors fixed: backbone capacity (ViT-L), evaluation data (Something-Something v2, SSv2), and classifier probe tuning. The four models span the three dominant self-supervised paradigms: joint-embedding latent prediction (VJEPA 2.1, V-JEPA 2), contrastive plus masked prediction (VideoPrism), and pixel reconstruction (VideoMAEv2). To test whether frozen representations remain competitive, the evaluation also compares frozen V-JEPA 2 against an end-to-end fine-tuned VideoMAE and a fully supervised TimeSformer. The evaluation utilizes Something-Something v2 (SSv2) for its rich class structure, which includes fine-grained human object interaction categories and pairs designed to force genuine temporal discrimination.

Feature Discriminability

The analysis of frozen video representations using global average pooled (GAP) features across the 30 selected action classes reveals that V-JEPA models produce the most discriminable representations. Specifically, V-JEPA models produce the most discriminable representations, achieving substantially higher linear probe accuracy than VideoMAEv2 and VideoPrism. Furthermore, V-JEPA separability is consistent across semantic categories, maintaining a uniform profile as difficulty increases from different-verb classes through same-verb pairs to pretend-vs-real pairs. This pattern suggests that joint-embedding predictive objectives capture higher-level semantic structure, whereas masked pixel reconstruction objectives remain more sensitive to low-level visual distinctiveness.

Corruption Robustness

Latent prediction models demonstrate superior resilience under pixel corruption compared to other paradigms. VJEPA 2.1 leads on five of six corruption types and shows the slowest accuracy degradation across severities. This resilience is attributed to the objective: the joint-embedding predictive objective discarding surface-level visual variation in favour of higher-order semantic structure. For instance, V-JEPA 2.1 achieves the highest accuracy retention on five corruption types, with elastic transform being the universal outlier that reduces all models to near-zero or single-digit retention. Crucially, while VideoPrism maintains near-perfect cosine similarity between clean and corrupted features even at the highest severity, it suffers from low accuracy retention, illustrating that Stable representations are not necessarily useful representations.

Fine-Grained Action Discrimination

Latent prediction models excel at capturing subtle physical cues without pixel reconstruction. VJEPA "outperforms the pixel-reconstruction baseline on virtually every class of ‘pretend actions,’ with the largest margins precisely on high-sensitivity classes whose discriminative signal is the absence of actual physical contact. This advantage is quantified across object size and detail sensitivity, as V-JEPA captures fine-grained contact cues despite never reconstructing pixels, whereas pixel reconstruction allocates capacity to surface texture, diluting the contact signal. This confirms that the latent predictive objective distils the spatiotemporal signatures that differentiate real from simulated contact."

Occlusion Robustness and Temporal Coherence

Latent prediction models exhibit a distinct advantage in handling missing input. VJEPA 2.1 is the most robust encoder for downstream decision making under occlusion, leading in probe accuracy across all three paradigms (Moving Block, Temporal Dropout, Spatiotemporal Patch Dropout). In contrast, VideoMAEv2 degrades the fastest under temporal dropout and patch dropout. Furthermore, latent prediction models internalize the arrow of time more strongly. Under video reversal, V-JEPA models achieve a Directional Semantic Coherence Score (DSCS) several times higher than VideoMAEv2 and VideoPrism, indicating that the latent space "acquires an oriented temporal axis separating pushing” and “pulling” by direction rather than collapsing them. This directional encoding is a signature of pretraining, as V-JEPA masks future patches to predict their latent representations from earlier context, creating an inherently directional learning signal.

Conclusion on World Modeling

The study concludes that across multiple robustness axes, latent prediction emerges as the paradigm producing a "uniformly strong representation.

Improvements for AI systems

Based on the systematic evaluation presented in this scientific paper, here are specific improvements that can be made to existing video world models by adopting the principles demonstrated by latent prediction models (specifically V-JEPA):

Improvement Strategy: Shift from Pixel Reconstruction/Contrastive Learning to Latent Prediction Objectives.

By fundamentally changing the pretraining objective from pixel reconstruction (like VideoMAEv2) or purely contrastive learning (like VideoPrism) to a joint-embedding latent prediction framework (like V-JEPA), the resulting AI system gains systemic robustness across five critical axes.

Specific Improvements and Capabilities of the Improved System:

Feature Improvement Mechanism Specific Capability Gained in the AI System

2.:---:---:---

3.0 Latent Prediction Objective Adoption (e.g., V-JEPA) over Pixel Reconstruction (e.g., VideoMAEv2). This involves training the model to predict a latent representation of future video frames rather than reconstructing raw pixels. The system learns to capture causal, high-order semantic structures and temporal dynamics in a compressed latent space, discarding surface-level visual noise. It will exhibit significantly superior performance under pixel corruption (motion blur, impulse noise) because it is not penalized for preserving corrupted pixel statistics that do not contribute to the underlying semantic prediction task.

4.0 Enhanced Corruption Robustness The latent representation remains stable even when input pixels are severely corrupted (e.g., elastic transform, severe noise). The model will maintain high classification accuracy under sensor noise and environmental changes where raw visual cues are unreliable, making it far more reliable for real-world deployment in unpredictable settings.

5.0 Superior Fine-Grained Discrimination The system learns to distinguish visually similar actions based on subtle physical contact cues (e.g., the difference between pushing and pulling or the absence of contact). It will accurately identify complex, fine-grained interactions (like detecting an empty grasp or subtle deformation) that require understanding the physics of interaction, rather than just recognizing dominant motion trajectories.

6.0 Internalization of Temporal Directionality The latent space is explicitly structured to encode the arrow of time, allowing the model to understand causality and directionality (e.g., pushing vs. pulling). The model will exhibit coherent semantic flips under video reversal, correctly predicting semantically antonymous actions (e.g., correctly identifying a push as a pull), which is critical for planning and understanding physical dynamics over time.

7.0 Robustness to Occlusion and Missing Input The system generalizes better when frames are missing or occluded because the latent prediction task forces the model to infer what should follow based on context, rather than relying on local visual patches. It will maintain high decision-making accuracy even under severe spatiotemporal patch dropout, as its representation is more structurally stable than those based solely on geometric similarity (like VideoPrism).

8.0 Robustness to Temporal Disruption The system maintains structural coherence when the temporal order of frames is shuffled or reversed. It will possess a high Directional Semantic Coherence Score (DSCS), meaning its prediction changes are semantically meaningful and directionally consistent, unlike models that change predictions randomly under reversal.

In summary, the improved AI system would transition from being a clean-benchmark classifier to a truly robust world model. It moves beyond simple pattern matching to acquire an internal sense of physics and temporal causality, ensuring reliable decision-making in noisy, occluded, or reversed real-world scenarios.

Abstract

Self-supervised video models are increasingly framed as world models, yet they are still evaluated almost entirely on clean video and reported as a final task score, obscuring how their representations behave under the degraded and ambiguous conditions a deployed world model must handle. We present the first systematic study of this component, analyzing four matched-capacity frontier self-supervised learning models that are strong candidates for world-model encoders -- V-JEPA 2.1, V-JEPA 2, VideoPrism, and VideoMAEv2 -- across five robustness axes relevant to their deployment as video world models: feature discriminability, corruption robustness, fine-grained discrimination, occlusion robustness, and sensitivity to temporal direction. We run the study on Something-Something-v2 and repeat every axis that ports on the egocentric EGTEA Gaze+, where the model ordering is unchanged. Our results reveal a distinct and consistent profile for latent-prediction models across all five axes. They degrade more gracefully under pixel corruption, preserve usable class structure rather than mere geometric stability under occlusion, capture fine-grained physical contact cues without reconstructing pixels, and uniquely encode the arrow of time. We finally test whether these representation-level differences matter for downstream world modeling, pairing each frozen encoder with an identical action-conditioned predictor and planner in a simulated manipulation environment. Only latent prediction yields near-complete task success, remains effective under degraded observations, and transfers to a manipulation task for which the predictor was never trained. Our results provide concrete new evidence that latent prediction is a promising foundation for robust world modeling.