PlanWAM: Planning-Shaped Future Representations for End-to-End Autonomous Driving

arXiv:2610.11382 · cs.RO, cs.LG · Submitted 2026-10-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa, and with me are Dev and Taro, guest researcher.

Dev: Today's paper: "PlanWAM: Planning-Shaped Future Representations for End-to-End Autonomous Driving".

Rosa: The gist The authors propose PlanWAM, a Planning-Shaped World Action Model, which reformulates future modeling in end-to-end autonomous driving as a planning-oriented predictive representation learning problem to enable foresighted planning.

Dev: First, who's behind it and why it matters.

Paper summary: Dev: So we’ve touched on the high-level idea of PlanWAM, which is reformulating future modeling as a planning-oriented predictive representation learning problem. The thesis is that the future state representation should be shaped by trajectory planning objectives to retain information useful for making decisions.

Rosa: Right. They argue that existing methods usually define future representations through pixel-level reconstruction or image latent prediction, like DINO or JEPA, and those might not retain the specific info needed for trajectory decisions.

Taro: Their key contribution is this shift: instead of just predicting the future state, PlanWAM proposes letting the planning task shape that future latent representation directly.

Dev: They do this by using a Temporal Register Pyramid to compress historical observations in a way that keeps things relevant, and then using Hindsight-to-Foresight Distillation to train a prior branch that predicts this specific planning-shaped representation from just history.

Rosa: That distillation step is important because it allows them to convert the privileged hindsight they get during training into something deployable for foresight when the car is actually driving.

Taro: They show that this planning-shaped future latent representation serves as a context for Foresight-Conditioned Planning, expressed by p(τ1:T, zˆ+t vt−n:t, ct−n:t) = pθ(ˆz+t ht) pθ(τ1:T ht, zˆ+t) <ref:2610.11382#pg1,this planning-shaped future latent representation>.

Dev: The numbers they put out are quite compelling. They achieved ninety-three point eight PDMS and ninety point nine EPDMS on NAVSIM-v1/v2 navtest, and reached a thirty-eight point seven HD-Score in zero-shot closed-loop evaluation on HUGSIM across different scenario difficulties <ref:2610.11382#pg3,38.7 HD-Score in zero-shot closed-loop evaluation on HUGSIM>.

Rosa: And they back that up with ablation studies showing that generic future feature supervision only got ninety-three point five PDMS, so you can see the planning shaping actually lifted the performance to ninety-three point eight PDMS <ref:2610.11382#pg3>.

Taro: They also showed that using the predicted latent for both trajectory generation and selection together gets them to ninety-three point eight PDMS, which means using it for one or the other isn't enough on its own <ref:2610.11382#pg3>.

Conclusion: Dev: So, wrapping up the PlanWAM paper, it’s about taking future modeling in autonomous driving and making it explicitly planning-aware by shaping the future representation with trajectory objectives during training.

Rosa: The authors are proposing this as a way to get a more effective and deployable form of foresight for world action models because the representation preferentially encodes information relevant to the actual driving task.

Taro: What this means for real-world autonomy is that we stop relying on generic image predictions and start training models whose internal world model directly reflects what it needs to plan a safe, efficient path.

Dev: The authors’ work suggests that focusing on *which* future representation is most useful for planning rather than just how to predict the scene evolution is a key area of research for driving AI.

Rosa: It moves the focus from just prediction accuracy to decision relevance embedded directly into the model's perception of time and space.

Taro: The implication here is that we need representations that are inherently coupled with planning objectives, not just auxiliary supervision targets, for robust autonomous systems.

Dev: That’s the core idea behind PlanWAM: planning-shaped future representations provide an effective and deployable form of foresight for world action models.

Jinchang Xu, Hongda Yu, Fengwei Dong, Wenhui Huang, Xi Wei, Yongzhi Liu, Sunan Zhang, Jirao Wang, Chen Lv, Bingbing Li

Southeast University · NIO · Nanyang Technological University · Tongji University

cs.RO, cs.LG

Submitted: 2026-10-08

Updated: 2026-10-08

Project page: https://jinchan9.github.io/PlanWAM_project_page

The gist: The gist The authors propose PlanWAM, a Planning-Shaped World Action Model, which reformulates future modeling in end-to-end autonomous driving as a planning-oriented predictive representation

Key concepts

Planning-Shaped Future Representation
This is a method where the model's prediction for the future is specifically guided by trajectory planning goals. Instead of just predicting what might happen, it predicts a future state that is optimized to be useful for making good driving decisions, like avoiding obstacles or reaching a target.
History Compression Module (Eη)
This module takes long sequences of past driving observations and compresses them into a concise summary. It does this in a way that prioritizes recent information, ensuring the model focuses on the most relevant historical context needed for current planning tasks.
Hindsight-to-Foresight Distillation (H2F)
This technique trains a prior branch to predict the planning-shaped future representation based only on history. It uses ground-truth future data during training to teach this branch how to create a useful, planning-relevant future prediction that can be used later for real driving scenarios.

Terminology

Summary

The gist The authors propose PlanWAM, a Planning-Shaped World Action Model, which reformulates future modeling in end-to-end autonomous driving as a planning-oriented predictive representation learning problem to enable foresighted planning.

How it works

  1. The core idea is to let the planning task shape the future-state representation so that it retains the information most useful for planning. A latent world model then predicts this planning-shaped future latent representation from historical observations and uses it for planning, enabling foresighted planning.

  2. The framework involves three main components: a history compression module Eη to compress multi-frame historical observations in a recency-aware manner. A privileged future posterior branch Pϕ observes ground-truth future frames and shapes its future latent representation with trajectory-planning objectives to obtain a planning-shaped future latent representation.

  3. Hindsight-to-Foresight Distillation trains a prior branch Pθ that depends only on history to predict this planning-shaped future representation. This converts privileged hindsight during training into deployable foresight at inference time.

  4. The predicted future latent representation serves as planning context and guides trajectory generation and selection. This results in Foresight-Conditioned Planning where the process is expressed as p(τ1:T, zˆ+t vt−n:t, ct−n:t) = pθ(ˆz+t ht) pθ(τ1:T ht, zˆ+t).

Key Contributions and Results

"We reformulate future modeling in end-to-end driving as a planning-oriented predictive representation learning problem, where the future representation is shaped directly by trajectory planning objectives rather than predefined reconstruction or generic prediction objectives, so that it preferentially encodes decision-relevant future information"

"PlanWAM achieves 93.8 PDMS / 90.9 EPDMS on NAVSIM-v1/v2 navtest and reaches 38.7 HD-Score on closed-loop HUGSIM in a zero-shot setting, demonstrating leading planning performance across both open-loop and closed-loop evaluations"

planning-shaped future representations provide an effective and deployable form of foresight for world-action models

Ablation Studies

Table 4: Overall component ablation of PlanWAM. ID TRP H2F FCP NC↑ DAC↑ TTC↑ C↑ EP↑ PDMS↑

The ablation study shows that adding the three core components in sequence, the Temporal Register Pyramid (TRP), Hindsight-to-Foresight Distillation (H2F), and Foresight-Conditioned Planning (FCP), leads to a PDMS of 93.8.

Table 5: Ablation of Future Representation. Null future – 93.4 Generic future latent Lfeature 93.5 Planning-shaped latent (ours) Lplan + Lscore 93.8

Generic future-feature supervision reaches only 93.5 PDMS, so a generic prediction target does not by itself yield planningrelevant information. Shaping the future latent directly with the trajectory planning objectives raises performance to 93.8 PDMS.

Table 6: Ablation of Future Latent Usage. Auxiliary supervision only × × Generation only × Selection only × Generation + Selection 93.8

Using the predicted latent for both trajectory generation and selection attains 93.8 PDMS.

Conclusion

PlanWAM achieves superior performance on both open-loop and closed-loop evaluations, demonstrating that planning-shaped future representations provide an effective and deployable form of foresight for world-action models.

Appendix Details

Appendix A gives the data flow and tensor shapes of the three transformers, for checking the implementation

The appendix details include a detailed architecture and data flow.

Appendix B gives the two-stage optimization and the full set of losses

The appendix also provides information on training objectives and optimization schedule.

Appendix C gives the metric definitions for NAVSIM-v1/v2 and HUGSIM, including how HD-Score and the HUGSIM average are computed

The appendix includes metric definitions for evaluation protocols.

Appendix D gives four additional ablations: posterior registers, future duration, the distillation weight, and which modules are trained

Additional controlled ablations are reported on NAVSIM-v1 navtest.

Appendix E gives additional visualizations of NAVSIM candidate trajectories and HUGSIM closed-loop rollouts

Additional qualitative results include visualizations of candidate trajectories and closed-loop rollouts.

The paper's experiments show that planning-shaped future representations improve trajectory generation and selection while reducing safety-critical failures, including collisions and low-TTC events. The zero-shot closed-loop generalization on HUGSIM across varying scenario difficulty demonstrates strong performance.

Improvements for AI systems

  1. The system can achieve foresightful trajectory planning by learning which future representation is most useful for planning, as opposed to generic representations like pixel reconstruction or image latent prediction, which are described as being mismatched with the downstream planning objective.

  2. The improved system can generate a compact history representation through the Temporal Register Pyramid, which is described as compressing historical scene registers in a recency-aware manner to retain dynamic cues from multi-frame, multi-view observations that are essential for future reasoning and planning.

  3. The system can obtain a planning-shaped future latent representation by directly conditioning it on trajectory planning objectives: the posterior world model WMpost ϕ aggregates historical evidence and ground-truth future information to obtain the posterior future latent z+t: z+t = Pϕ(ht, f + t).

  4. The system can enable deployable foresight at inference time by using Hindsight-to-Foresight Distillation to train a prior branch conditioned only on history to predict this representation, converting privileged hindsight during training into deployable foresight at inference time.

  5. The improved system can improve trajectory generation and selection by fusing the predicted future representation with the history representation as a foresight planning context to guide planning, as shown in the formulation: τ1:T ∼ πθ(ht, zˆ+t).

Sources

Related papers