GlanceWAM: Sparse Test-Time Imagination for World-Action Models

arXiv:2608.23927 · cs.CV, cs.AI, cs.RO · Submitted 2026-08-25 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "GlanceWAM: Sparse Test-Time Imagination for World-Action Models".

Tom: Video generative models offer rich physical priors for robot learning, yet existing world-action models (WAMs) face a fundamental trade-off: synchronous video generation at control rate is latency-prohibitive,

Jane: First, who's behind it and why it matters.

Title and authors: Tom: So, we’re diving into GlanceWAM: Sparse Test-Time Imagination for World-Action Models, which sounds pretty technical but really gets right to the core of how robots learn from video data. The paper introduces this idea of sparse test-time imagination as a way to improve what we call world-action models.

Jane: That title tells us a lot about the trick they’re pulling—it’s not about generating everything all at once, but rather imagining just a few key frames sparsely when needed during testing. It suggests that having this ability to look ahead in time, even if it's infrequent, can actually lead to better performance than models that try to do it continuously.

Lu: The authors are bringing together concepts from different architectures here, specifically showing how you can unify video world modeling and action policy learning within one unified video DiT using a dual-timescale formulation as the mechanism for this imagination.

Meng: When we look at the summary, they’re addressing that core dilemma where synchronous generation is too slow and abandoning test-time imagination hurts task success; GlanceWAM claims to solve both problems simultaneously by generating lookahead frames off the critical path.

Lalam: This decoupling of timescales is a powerful concept because it allows the system to handle different time resolutions—a slow clock for dreaming up a future goal and a fast clock for actually executing the current action—which feels like it’s going to make our AI systems much more flexible.

The paper's summary: Tom: The summary explains that GlanceWAM unifies video world modeling and action policy learning into one video DiT backbone, using a dual-timescale setup where the lookahead frame generation happens on a slow clock, around three seconds per frame. Meanwhile, the action head decodes the actual movement commands at the control rate of forty-eight milliseconds per chunk purely in latent space without blocking anything.

Jane: That’s a very clear way to put it: imagine you have one part of your system running slowly to visualize what might happen later, and another part running quickly to decide what to do right now, and they don't interfere with each other. It’s about pipelining the imagination off the critical path.

Lu: The paper points out that this is enabled by a non-interfering attention mask, which isolates the video representations so that those lookahead tokens don't contaminate what we use for observing the current state of things during action decoding.

Meng: From an engineering standpoint, having that isolation mechanism is key; it means we don't have to worry about our long-term imagination process messing up the short-term observation encoding, which is a major hurdle in these complex models.

Lalam: It’s impressive how they manage this complexity; they use staleness-robust horizon training to supervise the policy across varying time offsets, which means the model learns to handle those asynchronous lookaheads without collapsing.

The paper's improvements: Tom: The improvements section highlights how GlanceWAM achieves its results by using two specific loss functions during training. First, a video objective supervises forward dynamics velocity prediction on the noisy future target z(τ) t+Hf with a specific loss formula.

Jane: That first objective is focused entirely on making sure the visual world model accurately predicts the movement of that future target over a horizon of Hf, which essentially grounds the slow imagination in physical dynamics.

Lu: Then, they have an action head objective that optimizes an inverse dynamics objective to regress the continuous action chunk at forty-eight milliseconds, conditioned on the DiT's visual representations and crucially including that lookahead offset ∆.

Meng: That second loss function is where they show how the action head actually learns to use that lookahead information; it conditions its output on the visual context from both now and that future imagined frame.

Lalam: These training objectives are what allow the model to learn this complex relationship between slow, sparse foresight and fast, real-time control decisions, which is a major step forward in how we train these systems.

Conclusion: Tom: So to wrap up GlanceWAM: they’ve shown that by decoupling imagination from control using this asynchronous approach, the model can achieve state-of-the-art success rates on tasks like RoboCasa kitchen, reaching seventy-two point two percent, which is significantly better than synchronous baselines.

Jane: That result really hammers home the point that test-time visual imagination isn't just a nice feature; it’s causally load-bearing for success, and their approach lets you get that foresight without sacrificing your real-time speed.

Lu: The implication here is that we can build world models that are inherently better at planning because they aren't limited to just the immediate vicinity, which opens up possibilities for much more complex manipulation tasks.

Meng: From a deployment perspective, the fact that the lookahead generation runs on a background thread means our synchronous cost stays low at forty-eight milliseconds per chunk, which makes real-time operation feasible on actual hardware.

Lalam: This work suggests that by treating visual foresight as an asynchronous, latent-space conditioning mechanism rather than a synchronous bottleneck, we can create AI agents that are both fast and surprisingly capable of anticipating what needs to happen next.

Virginia Tech University · Drexel University

cs.CV, cs.AI, cs.RO

Submitted: 2026-08-25

Updated: 2026-09-29

Code: https://github.com/linhanwang/GlanceWAM

Importance score: 90/100

The gist: Video generative models offer rich physical priors for robot learning, yet existing world-action models (WAMs) face a fundamental trade-off: synchronous video generation at control rate is

Key concepts

Speed–Success Dilemma
Existing world-action models struggle because they try to predict the future and decide actions all at once. This causes slow performance or poor results: either the robot moves too slowly (high latency) or it fails because it cannot imagine what happens next during a quick action.
Dual-Timescale Formulation
GlanceWAM uses two different clocks to manage information flow. A slow clock handles generating 'lookahead' frames, which are used for planning. A fast clock handles the actual robot control, ensuring actions are executed quickly without waiting for the slower imagination process.
Asynchronous Latent-Space Inference
Instead of decoding future video frames immediately, GlanceWAM generates them into a compressed latent space. This lookahead information is then used later by the control system. This separation allows the high-speed control loop to run independently of the slower, computationally intensive frame generation process.

Terminology

Summary

Video generative models offer rich physical priors for robot learning, yet existing world-action models (WAMs) face a fundamental trade-off: synchronous video generation at control rate is latency-prohibitive, while abandoning test-time visual imagination sacrifices task success. This paper introduces GlanceWAM, which decouples imagination from control within a single video DiT to achieve both real-time inference and superior success rates by generating lookahead frames asynchronously off the critical path.

The Core Problem: The Speed–Success Dilemma

Existing WAMs couple future prediction and action decoding synchronously at the control rate, leading to prohibitive inference latency (1.1–3.8 s per chunk) and horizon degeneration because prediction is pinned to the short duration of an action chunk, causing world modeling to collapse into near-trivial observation reconstruction. This creates a dilemma: either synthesize video synchronously at the control rate and forfeit real-time reactivity, or discard test-time imagination and forfeit the performance benefits of visual foresight.

GlanceWAM Architecture

GlanceWAM unifies video world modeling and action policy learning within a single DiT backbone using a dual-timescale formulation. It decouples timescales by generating second-scale lookaheads off the critical path on a slow clock (Hf ≈ 3 s) while decoding action chunks in latent space at the control rate (48 ms). This is enabled by:

  1. A non-interfering attention mask (prefix-LM) that isolates video representations by preventing lookahead tokens from contaminating observation encodings.

  2. Staleness-robust horizon training that accommodates asynchronous lookahead aging, supervising the policy across varying time offsets (u ∼ U(0, Hf]).

Training and Loss Functions

The model is trained purely on demonstrations using a joint flow-matching objective:

  1. A video objective supervises forward dynamics velocity prediction on the noised future target z(τ) t+Hf:

Lvideo(θdit) = Eτ,ϵ,z vθ z(τ) t+Hf, τ z≤t, c − (ϵ − zt+Hf) squared.

  1. The action head optimizes an inverse dynamics objective regressing the continuous action chunk at ∈ R Ha×Da conditioned on the DiT backbone’s multi-layer visual representations h, task instruction c, and lookahead offset ∆:

Laction(θdit, ϕact) = Eσ,ϵa uϕa(σ) t, σ [hobs; hla], c, ∆ − (ϵa − at) squared.

Asynchronous Latent-Space Inference

At test time, the model executes a pure latent-space control path:

  1. Once per horizon Hf (∼3.0 s), the video DiT runs an ODE flow sampler to generate the lookahead latent zˆla from current observation tokens z≤t. Crucially, zˆla is never decoded to raw RGB pixels; it is retained entirely within the normalized latent space of the causal VAE, directly serving as slot-2 conditioning tokens for subsequent action forward passes.

  2. The action head decodes subsequent 0.8 s action chunks in real time (48 ms per chunk on a single NVIDIA A100 GPU), where each chunk conditions on the decaying lookahead offset ∆ = Hf − (t mod Hf) ∈ (0, Hf]. This allows the lookahead proposer to run concurrently at the Hf cadence without blocking the high-frequency control loop.

Empirical Results and Analysis

GlanceWAM achieves state-of-the-art performance, reaching 72.2% success on RoboCasa kitchen (surpassing synchronous Cosmos Policy at 67.1% and imagination-free co-training at 64.4%) and 99.0% on LIBERO, while executing at 48 ms per chunk (24× faster than synchronous baselines). Systematic diagnostics confirm that lookahead conditioning is causally load-bearing, as success drops from 71.5% to 61.6% when lookahead tokens are zeroed at test time. Furthermore, the action head actively incorporates lookahead representations into control decisions, with cross-attention mass concentrating on functional scene elements and displacement in predicted action chunks confirming active lookahead consumption during control. The framework is robust across sampling budgets, as success remains stable across 1 to 30 Euler steps (71.2% to 71.5%), indicating that the policy reads coarse where-to-go layout rather than high-frequency visual details.

Efficiency and Fidelity

The asynchronous design ensures that lookahead generation is pipelined on a background thread, meaning the synchronous cost per chunk remains at 48 ms, while the total refresh cost (≈37 ms per Euler step) never blocks the control loop.

Improvements for AI systems

Based on the provided scientific paper, here are specific, high-impact improvements for existing AI systems (specifically World-Action Models or WAMs) and what those improved systems can achieve:

  1. Improve Inference Latency in Real-Time Control Loops:

  2. Improve Task Success Rates Beyond Synchronous Baselines:

  3. Enable Long-Horizon, Distal Goal Planning in Manipulation Tasks:

  4. Develop Robust World Models for Unseen or Dynamic Environments (via Asynchronous Amortization):

5.1 Improving Inference Latency in Real-Time Control Loops:

The improved system can execute action decoding at a guaranteed, low frequency (e.g., 48 ms per chunk) while maintaining the capability to synthesize visual foresight that is seconds into the future. This is achieved by decoupling the generative glance (which runs on a slow clock off-path) from the fast control loop.

100% of action chunks are decoded in latent space in real-time, eliminating prohibitive diffusion sampling latency (which was 1.1–3.8 seconds previously). The system achieves this by holding a single lookahead latent frame for approximately four consecutive control cycles, effectively amortizing the high-cost visual synthesis over multiple actions.

5.2 Improving Task Success Rates Beyond Synchronous Baselines:

The improved system can achieve state-of-the-art success rates (e.g., 72.2% on RoboCasa kitchen and 99% on LIBERO) that surpass synchronous models (like Cosmos Policy at 67.1%) and imagination-free co-training (64.4%). This gain is attributed to the explicit spatial guidance provided by the lookahead conditioning channel, which teaches the policy where to go rather than just what is happening now.

5.3 Enabling Long-Horizon, Distal Goal Planning in Manipulation Tasks:

The improved system can effectively plan and execute long-horizon tasks (e.g., complex pick-and-place sequences or multi-step assembly) by conditioning the policy on a visual goal approximately 3 seconds into the future. This foresight allows for proactive movement and error correction based on anticipated distal subgoals, preventing horizon degeneration where models only focus on immediate surroundings.

5.4 Developing Robust World Models for Unseen or Dynamic Environments (via Asynchronous Amortization):

The improved system can operate in dynamic or uncertain environments by utilizing staleness-robust horizon training and asynchronous execution. This allows the model to remain functional even when the lookahead frame is generated asynchronously, and it provides a mechanism (the decaying offset) to ensure that future targets are relevant as the control loop progresses, making the world model more resilient to temporal delays in perception or generation.

Sources

Related papers