WNM-3D: A World Navigation Model with 3D Scene Conditioning for Closed-Loop VLN

arXiv:2608.07267 · cs.AI, cs.CV, cs.RO · Submitted 2026-08-20 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "WNM-3D: A World Navigation Model with 3D Scene Conditioning for Closed-Loop VLN".

Jane: The paper was written by Yuehao Huang, Yunzi Wu, Xiaotao Zhang, Xinhai Li, Jiankun Dong et al. from Institute of Artificial Intelligence, China Telecom and Zhejiang University and Tongji University and Shanghai Jiao Tong University.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper summary: Tom: We've been circling a big question in embodied eye, and this paper goes straight at it. Instead of just mapping language and camera frames to the next action, the model generates the future frames and the actions together, as one joint prediction. That single choice shapes everything else in the paper.

Jane: That's the bet that makes it a world model rather than a plain policy. The model has to produce a short video of what it expects to see, and a matching sequence of motions, and the two have to agree with each other. If the action says turn left but the imagined video shows the scene sliding sideways, the model has contradicted itself.

Tom: The three dee part is the conditioning. It takes the recent monocular RGB history — 33 frames — runs them through a frozen geometry encoder called VGGT-Ω, and a trainable adapter turns those geometry features into 450 tokens. Those tokens condition every future frame and action block throughout denoising.

Lu: I like that the geometry encoder stays frozen. The adapter is only about 24 million trainable parameters, and it's the piece that bridges the geometry representation into the diffusion transformer's token space. That decoupling means the upstream scene representation can evolve without retraining the whole world model.

Meng: Then there's the training side, which has three stages. First, supervised fine-tuning on A* expert demonstrations; second, DAgger, which collects expert corrections on states the policy actually visits; and third, DanceGRPO, an RL method for diffusion models using counterfactual rollout pairs.

Jane: Three stages — so the RL isn't the first thing you try.

Meng: Exactly. Each stage fixes a different failure mode, and the order turns out to be crucial, which we'll get to.

Tom: The numbers on the seen split jumped out at me, too. An 81 point 3 percent success rate and 78 point 3 SPL beats a strong baseline that even gets bird's-eye-view input. On unseen environments they still reach 46 point 8 percent success.

Jane: And the gain over their own 2D-conditioned control is about six points of success rate on seen environments. That isolates the value of the geometry conditioning pretty cleanly, since the control shares the same backbone and training recipe.

Lalam: What matters to me is the direction. You get a single generative system where visual foresight and action share the same geometry-aware context at inference time, and that pattern could carry over to other embodied tasks. The fact that geometry is used at inference, not just as a training crutch, is the part I find genuinely new.

Jane: Then there's the ablation, because DAgger turns out to be the dominant driver of the improvement. Applying the RL stage before DAgger actually makes things worse, which tells you the order of the training stages is doing real work.

Tom: So let's look at how they set up that claim. The first page spells out exactly why action-only training leaves a gap.

Page 1 of the paper: Jane: We're now on page one, and the argument opens with a precise complaint. Vision-language navigation models inherit strong semantic priors from pretrained vision-language models, but they're optimized mostly for action prediction. Nothing explicitly constrains how the visual observations should evolve under the predicted motion.

Tom: And that's consequential in continuous navigation, because every action changes the viewpoint. The evidence available for the next decision is literally created by the previous action. Action supervision alone doesn't capture that closed-loop dynamic, which is why the paper calls it a consequential omission.

Lu: The paper then frames a central question, and I think it's a sharp one. A monocular history contains multiple views of the same environment as the agent moves, so how do you recover geometry-aware information from that history and use it as shared context for both future-view prediction and action generation? The whole architecture is an answer to that question.

Meng: What I appreciate is that they don't invent a new sensor. The geometry comes from the RGB history you already have, through the frozen VGGT-Ω encoder, and the adapter converts it into a fixed-length prefix in the token space of the diffusion transformer.

Tom: And the three dee version's prefix is 450 tokens, exactly matching the length of the 2D control's VAE-encoded history. That's an important detail, because the comparison is fair by construction.

Jane: The mechanism is block-causal attention, where that clean prefix stays visible to every future video-action block. Within each block, visual and action tokens interact bidirectionally, while dependencies across blocks stay causal. So the geometric context guides both modalities throughout joint denoising, rather than being applied once at the start and then forgotten.

Lalam: They also list three contributions, and the list maps onto the rest of the paper. The geometry-conditioned formulation, the adapter design with its fusion and resampling stages, and the progressive training protocol with DAgger and DanceGRPO. Each one gets its own section and its own experimental evidence, which is a structure I appreciate.

Tom: Which brings us to where this fits in the literature. Page three positions the model among the existing world-action models and gives the architectural overview.

Page 3 of the paper: Tom: So page three opens with the related work, and the key contrast is with other world-action models. WAM-Nav, NavWAM, SWAM, WorldFly, WorldVLN — they all couple future-view prediction with action generation. None of them conditions that joint generation on geometry-aware representations recovered from the observed history.

Jane: There's also a subtle line within the geometry-aware group. DriveDreamer-Policy for driving, plus GeoSem-WAM and MECo-WAM, all bring geometry into world-action modeling. But the paper points out that MECo-WAM transfers the geometric prior through a training-time expert that gets removed at deployment, whereas this model uses the geometry as a shared inference-time condition. That's the key difference in how the geometric information is treated.

Lu: That distinction changes what the model can rely on. When a geometric expert disappears at deployment, the model has to make do without that information. Here, the scene tokens are part of every denoising step, so the geometry is actually load-bearing.

Meng: The method section then gives the architecture in one sweep. Both the three dee model and the 2D control share the same world-action backbone from DreamZero, the same block-causal attention, the same flow-matching objective, and the same three-stage training procedure. The only difference is where the history prefix comes from.

Jane: The 2D control is a good scientific choice. It uses the backbone's native VAE-encoded RGB history as its prefix, so when the three dee version does better, you can attribute the gain to the geometry-aware conditioning. Same prediction targets, same attention layout, same inference scheme.

Lalam: The serialization of the sequence makes the whole thing concrete. Clean prefix first, then the current frame as block zero, then future visual blocks and aligned action blocks, all under the block-causal mask. It's a compact way to define a joint prediction problem that has both a video stream and a control stream.

Tom: And the details of that joint prediction, the flow matching and the adapter internals, are what page five works through. That's where the mathematical structure becomes visible, and where the adapter's design choices get spelled out.

Page 5 of the paper: Jane: So we're on page five, and the math gets concrete. The paper adopts the joint flow-matching formulation from DreamZero, where each visual and action variable follows a linear path from noise to data. A shared diffusion transformer predicts the velocity field for both modalities.

Tom: One detail I found interesting is the coupling. A future visual block and its aligned action block share the same flow timestep, but they sample independent noise, so the pairing is in the schedule, not in the randomness. That keeps the two streams synchronized without forcing their noise to correlate.

Lu: Then the loss combines weighted velocity errors for video and action, with a mask on the padded action dimensions. The masking is a practical necessity because the backbone expects a fixed action width, and the physical action has only three components — two translations and a yaw. The mask simply zeros out the padding during training.

Meng: The adapter is the real meat of this page. VGGT-Ω produces patch features from four selected encoder levels, and the adapter fuses them with a gating network that's location-adaptive. Then it pools the fused memory onto a target lattice, adds learned slots and structured embeddings, and refines with two anchored deformable resampling layers and two factorized spatiotemporal blocks.

Jane: The deformable resampling is a clever way to stay efficient. Each target query predicts offsets and aggregation weights, retrieves a small neighborhood of source features around an anchor, and augments the content with source-coordinate embeddings. That avoids global cross-attention over the full source memory, which would be expensive.

Tom: And the output is always the same interface: 450 tokens at the transformer's hidden width. That fixed-length prefix is what decouples the geometry encoder from the generator, and it's what lets them swap in the 2D control without touching the rest of the system.

Lalam: So the architecture side is settled by page five. What remains is the question of how you train a model like this in the closed loop, which is a different kind of problem from offline fitting. Page seven covers that.

Page 7 of the paper: Tom: Page seven walks through Stage III, the DanceGRPO refinement, and it's the most intricate part of the pipeline. They partition the denoising transitions into four strata — the early steps, then step six, step ten, and the final steps. At each optimization step they pick one transition from each stratum.

Jane: Then comes the counterfactual trick. For each conditioning instance and stratum, they generate two branches that share the initial latents and all noise increments except at that single intervened transition. Both branches complete the full rollout, but gradients are replayed only through the intervention, which localizes the credit assignment.

Lu: The advantage calculation follows from that pairing. Each reward stream is standardized within the pair, so the advantage reflects the ordering between the two branches rather than the absolute reward values. Non-tied advantages land near plus or minus one over root two, which makes it a rank-style signal.

Meng: And the routing is careful too. The visual reward goes through the visual likelihood ratio, while navigation and stopping rewards go through the action-side ratios, because the visual and action SDE transitions use independent Gaussian noise. So it's a modality-routed surrogate objective rather than an exact likelihood ratio for the full joint transition.

Jane: The reward definitions are heavy, honestly. The visual reward combines pyramid SSIM with reconstruction and temporal consistency terms, plus a flow-action consistency bonus that compares the executed action block's displacement to camera motion inferred from the generated frames. The navigation reward covers geodesic progress, path-length agreement, goal potential, collisions, and route adherence.

Tom: What I take from this page is that getting RL to work on a diffusion policy requires a lot of careful engineering. There's a stopping reward with potential-energy terms, hard success and failure values, and an exit penalty, plus separate clipping thresholds for the visual and action ratios. That's the price of making the closed loop work.

Lalam: And that engineering is what gets tested in the experiments. Page nine lays out the benchmark, the baselines, and the flow-action consistency metric they designed to measure whether the imagined video matches the executed motion.

Page 9 of the paper: Jane: So page nine sets up the experiments, and the benchmark is GN-Bench. The seen split has a thousand episodes, the unseen split has five thousand, and they report the standard navigation metrics: navigation error, oracle success, success rate, trajectory length, and SPL.

Tom: The baseline list is meaningful. CMA, NaVid, UniNaVid, InternNav, GN-BAE — some use depth, one uses bird's-eye-view, and the strongest prior method reaches around 58 point 6 percent success on seen with BEV input. The 81 point 3 percent from this paper's model is a large jump on top of that.

Lu: Implementation details matter here. Both variants initialize from Wan2 point 2-TI2V-5B, use 33 history frames, predict four blocks of eight actions each, and run the world-action backbone at 160 by 320 while VGGT-Ω sees a 512 by 512 input. Stage one uses 16K A* demonstrations, and the DAgger datasets come to around 633K chunks for the three dee variant.

Meng: The flow-action consistency metric is the novel evaluation piece. It takes the visual-action block used for receding-horizon execution, computes optical flow on the generated frames, and maps flow descriptors to camera motion with a ridge regressor calibrated on 480 ground-truth clips. Then it compares that inferred motion to the cumulative XY displacement of the predicted action block, with a confidence weighting from forward-backward flow consistency.

Jane: They report three quantities from that protocol: the consistency score, a motion-magnitude error, and an action-side reward. All of it is evaluated on a fixed set of near-goal stops so that checkpoints from different training stages are compared on identical states. That fixed-set design is what makes the stage-wise comparison trustworthy.

Tom: And the results of that comparison, along with the navigation ablations, are on page eleven. That's where the training curriculum gets tested stage by stage, and where the design choices either pay off or don't.

Page 11 of the paper: Lu: Page eleven has the ablation that ties the whole story together. For the three dee model, stage one supervised training gets 49 point 6 percent seen success. Adding DAgger nearly doubles it to 80 point 6, and DanceGRPO pushes it to 81 point 3. The same pattern holds on unseen, from 39 point 7 to 45 point 7 to 46 point 8.

Tom: The striking result is what happens without DAgger. Applying DanceGRPO directly after stage one drops seen success from 49 point 6 to 39 point 4, and the 2D variant fails the same way. The paper's hypothesis is that the stage-one policy induces a narrow, error-prone distribution, so group-relative optimization can only rank candidates within that limited support.

Meng: That's a convincing story. DAgger first expands the useful policy support by injecting expert-corrected trajectories at policy-visited states, and only then does the reward-based stage have enough behavioral diversity to produce meaningful rankings. They're honest that they don't directly measure the diversity mechanism, though, so it remains a hypothesis.

Jane: The flow-action consistency table reinforces the geometry story. Across all three stages, the three dee model beats the 2D one on the consistency score and on motion error — at stage three, the scores are 0 point 3781 against 0 point 3609, with lower motion error too. Interestingly, DAgger temporarily increases the motion error on the near-goal stop set, and DanceGRPO then brings it back down below the stage-one level.

Tom: So the two evaluations tell complementary stories. Navigation metrics say DAgger is the workhorse, while consistency metrics say geometry-aware conditioning improves the alignment between what the model imagines and what it executes. The RL stage refines that alignment further, and the consistency advantage of the three dee model grows across stages.

Lalam: And the limitations are stated plainly. The consistency analysis covers only XY motion in the executed block, on near-goal states, and doesn't assess yaw or full perceptual fidelity. That restraint makes the positive results easier to trust.

Jane: That honesty carries into the conclusion, where they sum up what they've shown and what remains open. Let's close the discussion there.

Conclusion: Tom: So we land at the conclusion. The paper delivers a geometry-conditioned generative world-action model for continuous vision-language navigation, with a scene-to-token adapter that turns monocular history into shared context for both future frames and actions. The three-stage training recipe — supervised, then DAgger, then RL — is what makes the closed loop actually work.

Jane: And the evidence supports each design choice. Geometry conditioning beats the 2D control on navigation and on flow-action consistency, DAgger provides the dominant closed-loop improvement, and the reward stage refines the policy after the support has been expanded. Each stage in the curriculum earns its place.

Lu: The ablation is the part I'll remember. It shows that reward-guided optimization can actively hurt when applied too early, and that expert correction on policy-visited states is what unlocks it. That's a lesson that transfers well beyond navigation research.

Lalam: For the bigger picture, this is another sign that generative world models are becoming practical for control. The model doesn't just decide; it imagines the consequences of its decisions in pixel space, and it uses geometry to keep that imagination coherent with the actions. I expect that pattern to spread.

Meng: The remaining gaps are clear enough. The consistency evaluation is limited to XY motion near goals, the method relies on simulator rollouts for training, and scaling to longer horizons and harder environments stays open. The gain on unseen scenes is also smaller than on seen ones, so scene-level generalization is far from solved.

Jane: But the seen-to-unseen gap isn't a reason to dismiss the approach. The geometry advantage holds on both splits, the training recipe is stage-wise interpretable, and each component can be studied separately. That's a solid foundation for follow-up work.

Tom: Good place to stop. That was a dense paper, and we got to unpack it from the motivation all the way to the ablations.

Jane: Absolutely. We'll pick up the next one in just a moment.

Yuehao Huang, Yunzi Wu, Xiaotao Zhang, Xinhai Li, Jiankun Dong, Jiajun Lv, Chi Zhang, Chenjia Bai, Yong Liu, Xuelong Li

Institute of Artificial Intelligence, China Telecom · Zhejiang University · Tongji University · Shanghai Jiao Tong University

cs.AI, cs.CV, cs.RO

Submitted: 2026-08-20

Updated: 2026-08-21

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 75/100

The gist: WNM-3D: A World Navigation Model with 3D Scene Conditioning for Closed-Loop VLN Problem and Motivation.

Key concepts

World model
A model that predicts future observations and actions together, rather than just mapping inputs to actions. In this paper, it generates a short video of expected scenes and matching motions, requiring consistency between the two.
3D scene conditioning
Using geometry-aware features recovered from monocular RGB history to condition the generation of future frames and actions. A frozen encoder (VGGT-Ω) extracts geometry, and an adapter converts it into tokens that guide the diffusion transformer throughout denoising.
DAgger
A training method where expert corrections are collected on states the policy actually visits, rather than only on expert demonstrations. In this paper, it is a training stage that significantly improves performance, and its order relative to RL matters.
DanceGRPO
An RL method for diffusion models using counterfactual rollout pairs. It partitions denoising steps, generates paired rollouts with shared noise except at one intervened step, and computes advantages based on pairwise reward ordering, with modality-routed gradients.

Terminology

Summary

WNM-3D: A World Navigation Model with 3D Scene Conditioning for Closed-Loop VLN

Problem and Motivation. Recent vision-language navigation (VLN) systems increasingly adapt pretrained vision-language models (VLMs) into vision-language-action (VLA) policies that map egocentric observations and language instructions directly to navigation actions. However, such action-centric training does not explicitly model how the agent’s visual observations should evolve under its predicted motion. This is consequential in continuous navigation because every action changes the viewpoint and, consequently, the visual evidence available to subsequent decisions. Generative world-action models (WAMs) jointly predict future observations and actions, but existing WAMs for continuous VLN do not condition joint future-view and action generation on geometry-aware representations inferred from the observed history. A monocular observation history nonetheless contains multiple views of the same environment as the agent moves. The central question is how can geometry-aware information recovered from this history be incorporated as shared context for both future-view prediction and navigation-action generation?

Proposed Method. The paper presents WNM-3D, described as "the first generative world-action model for continuous VLN to use geometry-aware scene tokens inferred from the observation history as a shared clean inference-time condition for both future-view prediction and action generation." The architecture comprises three key elements:

  1. History-Prefix Construction: A frozen feed-forward geometry encoder extracts cross-view features from a history of monocular egocentric RGB observations (specifically VGGT-Ω). A trainable 3D Scene-to-Token Adapter converts these geometry-aware features into a fixed-length token sequence aligned with the hidden width and token layout of the world-action Diffusion Transformer (DiT).

  2. Joint Latent World-Action Modeling: The model builds on the joint video-action flow backbone of DreamZero, jointly predicting a finite-horizon future-view rollout and a temporally aligned navigation-action rollout via a shared world-action DiT.

  3. Block-Causal Self-Attention: "The clean history prefix remains visible to all temporal blocks; visual and action tokens interact within each block while attending only to the current and preceding blocks, preserving causality across the predicted horizon. Thus the scene context guides both modalities throughout joint denoising."

3D Scene-to-Token Adapter. The adapter converts geometry features into a fixed-length geometry-aware prefix through four stages: feature fusion, structured query formation, anchored deformable resampling, and factorized spatiotemporal refinement. Specifically: (a) Encoder-level feature fusion uses a lightweight gating network to assign location-adaptive weights across encoder levels to produce a shared geometry-aware source memory; (b) Query formation adaptively pool[s] the source memory onto a target lattice and combine[s] the resulting scene base with learned target-grid slots and structured embeddings; (c) Anchored deformable resampling uses R anchored deformable resampling layers where each target query is associated with a regular anchor mapped onto the source history–height–width lattice for fine-grained local retrieval; (d) Spatiotemporal refinement applies spatial self-attention within each temporal slice, followed by temporal self-attention at each spatial location and an MLP residual, with a detail head and a coarse head projecting to the DiT width. The configuration uses K=33 history frames at 512×512 for the geometry encoder, B=4 blocks with Nv=8 future frames and Na=8 actions per block, a target lattice of 9×5×10 (yielding Nc=450 prefix tokens), and 23.97M trainable adapter parameters.

Training Protocol (Three Stages). WNM-3D is trained progressively:

  • Stage I: Offline A SFT.* Supervised world-action fine-tuning on A*-generated demonstrations initializes joint future-view and action prediction using the world-action objective LWA (flow matching with modality-specific velocity fields).

  • Stage II: Closed-Loop DAgger-SFT. DAgger-style data aggregation queries the A* expert at policy-visited states and supplies corrective actions together with their trajectory-consistent future observations. Stage II datasets contain approximately 691K chunks for WNM-2D and 633K chunks for WNM-3D.

  • Stage III: Closed-Loop DanceGRPO. We apply DanceGRPO to the joint visual-action flow policy, using visual, navigation, and stopping rewards derived from simulator rollouts for closed-loop policy optimization. It uses stratified counterfactual rollout pairs with pairwise multi-stream credit assignment via a paired counterfactual rank advantage, with visual rewards routed through visual transition-density ratios and navigation/stopping rewards through action ratios. The complete Stage-III objective is LIII = Lv + λact(Lnav + λstop Lstop).

For comparison, the authors instantiate WNM-2D, "which shares the same world-action backbone, block-causal attention, prediction targets, and three-stage training procedure, but replaces the geometry-derived prefix with the backbone’s native VAE-encoded RGB-history prefix. Both variants initialize from Wan2.2-TI2V-5B and use receding-horizon inference: Only the first action block At,1 is executed, after which newly acquired simulator observations extend the observed trajectory before replanning."

Experiments and Results.

  • Closed-loop navigation on GN-Bench: "On the Seen split, WNM-3D achieves 2.0 NE, 87.2% OS, 81.3% SR, and 78.3% SPL. Relative to the strongest prior method in Table 1, which additionally uses BEV observations, this improves SR by 22.7 percentage points and SPL by 19.7 points. On the Unseen split, WNM-3D obtains 4.4 NE, 54.1% OS, 46.8% SR, and 43.5% SPL. Compared with the strongest FPV-only prior baseline, it improves SR by 7.9 points and SPL by 6.2 points."

  • Effect of geometry-aware conditioning: "With the same world-action backbone and three-stage training recipe, WNM-3D improves over WNM-2D by 5.7 SR and 5.4 SPL points on Seen environments. The corresponding gains on Unseen environments are 0.9 SR and 0.7 SPL points."

  • Training-stage ablation: "For WNM-3D, Stage II raises Seen SR from 49.6% to 80.6% and SPL from 49.1% to 77.9%. On Unseen environments, SR increases from 39.7% to 45.7% and SPL from 38.9% to 41.9%. DAgger therefore accounts for the dominant improvement." Starting from Stage II, DanceGRPO further improves WNM-3D by 0.7 SR and 0.4 SPL points on Seen environments and by 1.1 SR and 1.6 SPL points on Unseen environments. Crucially, "Applying DanceGRPO directly after Stage I degrades navigation for both variants... The ablation supports this interpretation, although direct measurements of within-group action diversity and reward spread would be required to verify the mechanism."

  • Flow-action consistency: On a fixed near-goal evaluation set, WNM-3D also achieves higher flow–action consistency and lower visual-motion error than the 2D-conditioned counterpart. Specifically, "WNM-3D achieves higher flow–action scores and lower motion errors than WNM-2D at all three evaluated checkpoints. Its flow–action score exceeds that of WNM-2D by 0.0151, 0.0157, and 0.0172 after Stages I, II, and III, respectively. WNM-3D also achieves lower motion errors by 0.0022, 0.0032, and 0.0034, together with higher action rewards at every stage. Both variants improve across stages, with DAgger substantially improv[ing] action quality and flow–action consistency, but temporarily increas[ing] the motion error on the near-goal STOP set, while DanceGRPO subsequently reduces the motion error below its Stage-I level while further improving both the consistency score and action reward."

Contributions. The paper lists three contributions: (1) We formulate a geometry-conditioned WAM for continuous VLN, in which geometry-aware scene tokens derived from observation history jointly condition future visual latents and temporally aligned navigation actions. (2) "We introduce a modular 3D Scene-to-Token Adapter that combines geometry-aware feature fusion, content-initialized target queries, anchored deformable resampling, and factorized spatiotemporal refinement to produce a fixed-length prefix compatible with the world-action DiT. (3) We establish a progressive closed-loop training protocol combining supervised world-action learning, DAgger adaptation to policy-visited states, and DanceGRPO refinement, and provide stage-wise analyses of navigation performance and flow–action consistency."

Limitations noted. The current consistency analysis is restricted to XY motion in the visual–action block used for receding-horizon execution and does not assess yaw or full perceptual future-view fidelity. The results also do not support a stronger claim of reduced Seen-to-Unseen degradation from geometry-aware conditioning, given the smaller Unseen gains.

Improvements for AI systems

  • The AI system can condition both future-view prediction and navigation-action generation on the same geometry-aware scene tokens derived from a monocular RGB observation history, so its visual forecasts and motor commands are jointly consistent with the 3D structure of the environment.

  • It can convert raw multi-view RGB history into a fixed-length, transformer-compatible geometry prefix via a 3D Scene-to-Token Adapter, enabling a frozen large geometry encoder to inject structured spatial knowledge into a pretrained world-action Diffusion Transformer without retraining the encoder.

  • The system can attend to the entire geometry-aware history prefix across all temporal blocks while preserving block-causal attention among predicted future views and actions, allowing long-horizon context to guide each step without leaking future information into past decisions.

  • It can jointly generate a finite-horizon rollout of future visual observations and a temporally aligned sequence of navigation actions using a shared flow-matching diffusion backbone, producing action plans that are explicitly tied to predicted visual motion rather than being decoupled from perception.

  • It can use anchored deformable resampling and factorized spatiotemporal refinement inside the adapter to retrieve fine-grained local geometry features from a compact target lattice, keeping the conditioning signal informative while remaining computationally tractable.

  • It can be trained in three progressive closed-loop stages: first supervised on A*-generated demonstrations; then refined with DAgger-style data aggregation at policy-visited states, which corrects actions where the agent actually goes; and finally optimized with DanceGRPO using stratified counterfactual rollout pairs and multi-stream visual/navigation/stopping rewards, improving both success rate and path efficiency.

  • After this training protocol, the system achieves a 22.7-point SR gain and 19.7-point SPL gain on seen environments over prior methods with BEV observations, and a 7.9-point SR gain and 6.2-point SPL gain on unseen environments over the strongest RGB-only baseline.

  • The system can improve navigation performance even when geometry conditioning is the only difference: with identical backbone and training, the geometry-conditioned model outperforms the 2D-conditioned variant by 5.7 SR and 5.4 SPL points on seen environments, and 0.9 SR and 0.7 SPL points on unseen environments.

  • It can quantify and improve flow–action consistency: the geometry-conditioned system consistently achieves higher flow–action scores and lower visual-motion errors across all training stages, meaning its predicted future frames match the motion implied by its executed actions.

  • It can use receding-horizon closed-loop inference: execute only the first predicted action, ingest new simulator observations, and replan, so the policy continuously corrects itself from real visual feedback rather than committing to a long open-loop plan.

  • The same architecture and training pipeline can be instantiated with or without geometry conditioning, allowing controlled evaluation of how much 3D-aware contextualization contributes to embodied navigation and future-state prediction.

  • The improved system can therefore serve as a scalable generative world-action model for continuous vision-language navigation, enabling robots and embodied agents to plan visually coherent action sequences in novel environments, recover geometric context from egocentric video, and reliably improve through closed-loop self-training.

Sources

Related papers