Modality-Autoregressive World-Action Models

summary

The gist

The gist The ModAR model introduces a world-action model that autoregressively denoises multiple future-observation modalities before predicting actions, and it outperforms prior WAM formulations.

In short

ModAR is a World-Action Model that predicts future observations as multiple modalities by autoregressively denoising them sequentially before predicting actions. It outperforms existing models by conditioning each prediction on previously generated modalities, achieving the highest success rates across data scales and showing improvements with actionless data.

Key concepts

Modality-Autoregressive World Modeling: This approach involves predicting future observations (like images or depth) as a sequence of different types. The model generates these predictions one after another, where each new prediction is refined by using the clean version of the previous ones. This allows for a more structured and detailed generation process.
Block-Causal Modality Generation: The model predicts modalities in a specific order: point tracks first, then DINO features, then depth maps, and finally RGB images. This sequence is chosen because simpler, structured representations are predicted before more complex ones. Each prediction step relies on the successful generation of the preceding ones to ensure coherence.
Inverse Dynamics Model (IDM): After predicting future observations, the model uses an IDM to determine the robot's actions. This step maps the generated future observations directly into physical movements that would cause those observed transitions. It ensures that the predicted visual sequence is physically plausible for robotic control.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Federated Learning — explained in 16 episodes
Retrieval-Augmented Generation (RAG) — explained in 16 episodes
Interpretability — explained in 15 episodes
Sample Complexity — explained in 15 episodes
Generalization — explained in 13 episodes

This episode discusses

The paper

Modality-Autoregressive World-Action Models · Read on arXiv

Adam Hung, Bardienus P. Duisterhof, Deva Ramanan, Jeffrey Ichnowski

Carnegie Mellon University

World-action models (WAMs) jointly model future observations and actions, typically predicting the future as RGB images. Other visual modalities such as depth, pretrained visual features, and point tracks can more efficiently capture geometric, semantic, and motion features. However, how best to combine these modalities within WAMs remains an open question. We introduce ModAR, the first WAM to autoregressively denoise multiple future modalities before predicting actions. This allows each prediction to condition on previously generated modalities. We train from scratch to systematically study how training-data mixtures, predicted modalities, and WAM formulations affect performance. In our evaluations, WAMs benefit from predicting point tracks, DINO features, and depth maps, while additionally predicting future RGB does not provide a consistent benefit. We also find that ModAR's sequential generation outperforms existing WAM formulations, with the highest average success rate at all evaluated data scales. We also fine-tune the video-model-initialized WAM Flex- π on the same data; ModAR achieves a slightly higher observed average success rate (75% vs. 72%) while using approximately 20 times fewer training FLOPs and no pretraining. On three real-world bimanual tasks, ModAR outperforms baselines and improves with human videos.

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "Modality-Autoregressive World-Action Models".

Dev: The gist The ModAR model introduces a world-action model that autoregressively denoises multiple future-observation modalities before predicting actions, and it outperforms prior WAM formulations.

Rosa: First, who's behind it and why it matters.

Paper summary: Rosa: Now, let's break down what this paper is actually proposing with "Modality-Autoregressive World-Action Models."

Dev: So, at its core, these models usually take future observations and actions and try to predict them as RGB images. This paper argues that we can do better by using other visual modalities like depth or point tracks alongside those RGB predictions.

Rosa: The thesis of ModAR is that it predicts the future as a sequence of multiple modalities, and it does this autoregressively, meaning one modality is predicted at a time, conditioned on the ones already predicted.

Taro: So imagine the robot first predicts where things are in terms of their tracks, then uses that to predict what those things look like with DINO features, and then uses that depth information for geometry.

Dev: That's right. The architecture they use is a shared diffusion transformer where these different modalities get processed by modality-specific experts before combining into a common token space.

Rosa: They also have this sequence of prediction steps where the final robot action prediction step acts like an inverse dynamics model, taking those generated future observations and mapping them to the actions that cause those transitions.

Taro: The generation order they chose is tracks, then DINO, then depth, and finally RGB because they intuitively felt that simpler, more structured representations are easier to predict towards the more complex ones.

Dev: This specific ordering is what they found was key to making the sequential conditioning work well within their model design.

Rosa: They train everything from scratch specifically to study how mixing training data, using these predicted modalities as context, and changing the world-action model formulation impacts the final performance metrics.

Taro: So it’s not just about getting a high number; it’s about systematically understanding which components of the prediction pipeline contribute most to success on real tasks.

Dev: And they found that this modality-autoregressive generation method performs best across all the different data scales they tested, and benefits most from having extra actionless data.

Rosa: So, if you're wondering why this matters, it’s because it shows that using these multiple structured inputs sequentially is a promising path instead of sticking only to predicting future RGB images.

Conclusion: Dev: So looking at the "Modality-Autoregressive World-Action Models" paper by Hung et al., the main point is this model uses sequential prediction for world-action modeling.

Rosa: Exactly. It predicts multiple future observations—like depth, DINO features, and point tracks—in a specific order before finally predicting robot actions based on those predictions.

Taro: The authors are showing that this sequence of predictions helps the system condition each step on what it already knows from the previous steps.

Dev: They found that predicting these modalities sequentially, starting with tracks and ending with RGB, provides additive performance gains over other methods.

Rosa: What this means for someone just listening to the show is that for a robot to predict its next move in a complex environment, it's better if it looks at several different types of data points before deciding on an action.

Taro: It moves us away from thinking about just one monolithic prediction and towards building a pipeline where different kinds of information feed into each other sequentially.

Dev: They also noted that even when they removed the future RGB prediction, ModAR still performed better than other formulations when compared to models that use separate action models.

Rosa: Overall, this work suggests that using modality-autoregressive prediction is a viable alternative for world-action modeling instead of relying solely on predicting the future RGB image.

More episodes

← Home