Teaching Video Generators to Remember: Eliciting Dynamic Memory for Out-of-Sight State Evolution

arXiv:2605.25333 · cs.CV · Submitted 2026-05-25 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Teaching Video Generators to Remember".

Jane: Video world models often fail to maintain evolving states when evidence is unobserved, leading to frozen or implausible resets upon re-observation.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Alright, moving on to the title and who came up with this work, "Teaching Video Generators to Remember: Eliciting Dynamic Memory for Out-of-Sight State Evolution." The authors are Tianshuo Xu, Yichen Xie, Depu Meng, Chensheng Peng, Quentin Herau, Bo Jiang, Yihan Hu, and Wei Zhan from Applied Intuition at UC Berkeley.

Jane: It’s interesting to see this coming from a group focused on applied intuition; it suggests they're looking for solutions to tangible problems in how these models operate in the real world rather than just theoretical constructs. The title itself really hammers home that the goal is teaching video generators to remember things that are out of sight.

Lu: The combination of authors, especially with Depu Meng involved, shows a strong link between deep research and practical implementation, which I find very appealing for pushing the boundaries of what these models can actually achieve in complex environments.

Meng: I'm curious about their approach because the problem they’re solving—maintaining state across interruptions—is something I see constantly in real-time systems where context window management is critical; how deep does this memory structure go before it starts becoming computationally prohibitive?

Lalam: It’s exciting to think about this level of memory capability; if we can give models that persistent world awareness, it could fundamentally alter how we design interactive experiences in virtual environments or even for long-form narrative content.

The paper's summary: Tom: So, the summary boils down to this: current video world models struggle to keep track of what’s happening when evidence is temporarily missing, which causes them to freeze or reset when they see the scene again. This paper proposes ReMind, a framework that treats the KV cache as dynamic memory and uses structured training and camera-aware addressing to allow for state evolution even when parts of the world are unobserved.

Jane: In simpler terms, it's about giving these generative models a way to remember things that happened previously, not just what’s immediately in front of them. Instead of just relying on the very last few frames, ReMind sets up mechanisms to pull in relevant historical data when the current visual context is incomplete.

Lu: The paper details how they construct training data around over one hundred dynamic events, turning real videos into a frame graph with protected anchors and explicit temporal gaps, which helps define what these key memory points are.

Meng: That structured data construction sounds intensive; I’m picturing the pipeline required to process all that event information just to prepare the training set for this kind of memory retrieval. What does that look like in terms of engineering overhead?

Lalam: The concept of event-aware training is very powerful because it forces the model to pay attention specifically when those important state transitions happen, which could lead to much more robust and consistent outputs overall.

The paper's improvements: Tom: Now we get into how they actually improve things. They introduce a few key components, like Projective Memory Rotary Positional Embedding, or PM-RoPE, which is a camera-phase extension to the standard RoPE. This mechanism lets the model retrieve historical anchors using a single attention operation instead of having to manage separate spatial and temporal routing paths.

Jane: That PM-RoPE sounds like a clever way to solve that decoupling problem between where things are spatially and when they happened temporally; it seems designed specifically to bypass some of the issues with standard dual-attention methods by unifying the address within one self-attention step.

Lu: They also use a very specific training curriculum, which includes regimes like node-drop and reference-cache training, all designed to force the model to ignore unreliable recent context and instead focus on recovering from surviving historical evidence when things get interrupted.

Meng: That node-structured curriculum sounds like a rigorous way to test the memory system; I wonder how they balance the different training regimes to ensure that the model learns both local continuity and this broader, more persistent state recovery simultaneously.

Lalam: The dynamic auxiliary loss, specifically the Dynamic Temporal-Delta Loss L∆, which adaptively adjusts its weight based on global spatial change, seems like a smart way to give a very targeted signal for when the model should prioritize recovering that hidden state information.

Conclusion: Tom: To wrap things up on "Teaching Video Generators to Remember: Eliciting Dynamic Memory for Out-of-Sight State Evolution," the authors show that ReMind’s dynamic non-local routing allows it to bridge temporal gaps, leading to a substantial improvement in state progress and physical coherence compared to other video models.

Jane: Essentially, they've managed to make the model better at maintaining a persistent world state across interruptions by re-conceptualizing the KV cache as dynamic memory and using specific training strategies tailored for event-aware retrieval.

Lu: The implication here is that we can move beyond systems that only see what’s immediately visible and start building generative systems capable of maintaining long-horizon physical consistency, which opens up a lot of creative avenues for simulating complex dynamics.

Meng: Practically, if this works as claimed, it suggests we could design more reliable simulation environments where agents don't lose track of their surroundings during moments of necessary occlusion or environmental change.

Lalam: For me, the biggest takeaway is how this diagnostic lens helps us understand exactly when our current visual models fail to maintain persistence; it gives us a way to diagnose those hidden-state evolution failures in media production.

Tianshuo Xu, Yichen Xie, Depu Meng, Chensheng Peng, Quentin Herau, Bo Jiang, Yihan Hu

Applied Intuition University of California, Berkeley

cs.CV

Submitted: 2026-05-25

Updated: 2026-09-29

Project page: https://remind-applied.github.io

Importance score: 92/100

The gist: Video world models often fail to maintain evolving states when evidence is unobserved, leading to frozen or implausible resets upon re-observation.

Key concepts

ReMind
A framework proposed in the paper that treats the KV cache as dynamic memory. It uses structured training and camera-aware addressing to allow video generators to remember things that happen when parts of the world are unobserved, enabling state evolution.
PM-RoPE
Projective Memory Rotary Positional Embedding. This is a camera-phase extension of standard RoPE that allows the model to retrieve historical anchors using a single attention operation, unifying spatial and temporal routing within one self-attention step.
Event-aware training
A training curriculum that includes regimes like node-drop and reference-cache training. This forces the model to ignore unreliable recent context and focus on recovering from surviving historical evidence when things are interrupted.
Dynamic Temporal-Delta Loss L∆
A dynamic auxiliary loss that adaptively adjusts its weight based on global spatial change. It provides a targeted signal for when the model should prioritize recovering hidden state information.

Terminology

Summary

Video world models often fail to maintain evolving states when evidence is unobserved, leading to frozen or implausible resets upon re-observation. This gap between generating visually plausible local motion and preserving persistent world state across interruptions exposes a critical failure in current video generators. The paper introduces ReMind, a framework designed to elicit dynamic memory behavior from pretrained autoregressive video diffusion transformers by treating the KV cache as dynamic memory, thereby enabling out-of-sight state evolution through structured training and camera-aware addressing.

ReMind Framework Overview

ReMind is a framework that elicits dynamic memory behavior from a pretrained causal video diffusion transformer by combining memory-oriented data construction, event-aware training, and pretraining-compatible cache adaptation. The core idea is to move beyond treating the KV cache as mere recent rollout context or a systems resource. Instead, ReMind reconceptualizes the KV cache as dynamic memory, establishing adaptive grounding via PM-RoPE to dynamically retrieve the most relevant historical anchors for continuous state evolution.

Memory-Oriented Data Construction

The framework constructs training data around a taxonomy of over 100 dynamic events, which are converted into a frame graph with protected anchors, degraded intervals, and explicit temporal gaps. This construction involves:

  1. Using VLM-filtered real videos to capture visible state evolution.

  2. Augmenting clips with interruption augmentations inspired by STEVO-Bench, including camera loops, light toggles, moving occluders, and zoom/camera perturbations.

  3. Deriving event nodes from loop-return points or known peak/recovery frames to define the protected anchors and recovery regions used by the memory curriculum.

Projective Memory Rotary Positional Embedding (PM-RoPE)

To make retrieval geometrically meaningful, ReMind introduces PM-RoPE, a camera-phase extension to rotary position embedding (RoPE). This mechanism solves the problem of decoupling spatial and temporal routing. Unlike dual-attention mechanisms that add separate spatial branches, PM-RoPE grants cached entries unified spatiotemporal addresses within a single self-attention operation, which is achieved by injecting camera-conditioned phase offsets into the standard RoPE path. This allows the model to bypass the Markovian temporal penalty and retrieve correct historical anchors at a single-attention cost.

Node-Structured Training Curriculum

ReMind trains the model with a node-structured curriculum designed to force retrieval of reliable state anchors rather than relying on local continuity. The four complementary regimes include:

  1. Node-drop training, which corrupts past chunks with high noise timesteps or replaces interruption nodes with pure-noise latents, forcing the model to ignore unreliable recent context and recover from surviving historical evidence.

  2. Noisy memory training, which corrupts past chunks with high noise timesteps while preserving at least one event anchor.

  3. V2V frontier training, which keeps a clean or degraded prefix and supervises only the recovery suffix.

  4. Reference-cache training, which prepend[s] clean reference chunks from the undegraded clip at old positions and starts the target video after a sampled temporal gap, directly teaching retrieval from non-contiguous memory.

Dynamic Auxiliary Loss and Evaluation

To further encourage dynamic memory exploitation, ReMind uses a novel strategy involving a flow-matching objective augmented with an auxiliary penalty. The Dynamic Temporal-Delta Loss L∆ is enforced to match ground-truth inter-frame changes, and its weight is adaptively adjusted: λadapt = α · exp(−γ · σbatch), where λadapt boosts the delta loss when global spatial change is small, providing a strong optimization signal for hidden state recovery. Evaluation uses STEVO-Bench for targeted hidden-state recovery and VBench for general I2V quality, demonstrating that ReMind achieves the best overall score among compared models on these tasks.

Key Findings

Experiments show that ReMind's dynamic non-local routing allows it to bridge temporal discontinuities, yielding a substantial improvement in state progress and physical coherence compared to other video models. Furthermore, KV-importance maps verify that the model breaks the recency bias, attending heavily to pre-occlusion anchors when recovering from occlusions. Ablation studies confirm that the combination of PM-RoPE and dynamic loss leads to superior performance in both low-level perceptual similarity (LPIPS) and high-level semantic consistency under degradation.

Limitation

The paper notes a limitation: ReMind focuses on specific interruptions like occlusion, darkness, lookaways, and loop closures and does not solve all physical reasoning failures. Additionally, the pipeline's reliance on camera and depth quality means pose errors in web videos may weaken PM-RoPE supervision.

Broader Impact

ReMind offers a diagnostic lens for hidden-state evolution, helping identify when visually plausible models fail to maintain persistent world states.

Improvements for AI systems

Here are specific improvements that can be made to AI systems by implementing the ReMind framework, and what those improved systems will be capable of:

  1. Improve persistent state tracking in video generation models (e.g., Diffusion Transformers) during unobserved periods (occlusion, darkness).

  2. Enable video generation models to maintain physically plausible object states across long-term interruptions that exceed the current visual context window.

  3. Develop generative systems capable of memory recall or state recovery when a scene is temporarily obscured or when external conditions (like light toggles) change, ensuring the final output reflects the state evolution that occurred before the interruption.

  4. Enhance semantic consistency and temporal coherence in generated video sequences by forcing models to retrieve relevant historical visual anchors rather than relying solely on recent, potentially corrupted context.

  5. Increase robustness against catastrophic forgetting when training models for new dynamics or conditions, as the memory-elicitation curriculum prevents the model from overwriting essential prior knowledge.

  6. Enable fine-grained control over physical state evolution in generated videos by allowing the system to explicitly choose which historical observation (e.g., a clean anchor vs. a corrupted frame) is most relevant for the current generation step, guided by camera pose and event type metadata.

Specifically, the improved AI system (ReMind) can perform:

  1. Generate realistic long-horizon videos where objects continue their motion or state changes even when the camera is momentarily blocked or the lighting shifts drastically.

  2. Produce physically consistent outcomes in complex scenarios, such as a pedestrian resuming a walk after being temporarily hidden by an occluder, maintaining the correct walking velocity and direction based on prior observation.

  3. Execute state recovery tasks where the model must correctly resume an object's state (e.g., a pouring container continuing to fill) after a 20-second period of darkness, accurately reflecting the volume accumulated during that dark interval.

  4. Achieve superior performance on benchmarks like STEVO-Bench by demonstrating high State Progress and Physical Plausibility, specifically by successfully navigating temporal discontinuities.

  5. Synthesize high-quality video content from static images (Image-to-Video) while maintaining strong semantic accuracy and motion quality, even when the generation process involves complex dynamic events that require historical context for accurate rendering.

Sources

Related papers