LiteNWM: Efficient Latent World Models for Onboard Visual Navigation in the Wild

arXiv:2610.12368 · cs.RO · Submitted 2026-10-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "LiteNWM: Efficient Latent World Models for Onboard Visual Navigation in the Wild".

Dev: The gist The LiteNWM model selects trajectories from a frozen policy without RGB reconstruction by combining action-conditioned multi-horizon latent prediction with incumbent-relative scoring How it works LiteNWM is designed to…

Rosa: First, who's behind it and why it matters.

Paper summary: Rosa: So we're looking at this paper called LiteNWM: Efficient Latent World Models for Onboard Visual Navigation in the Wild, and essentially what it’s claiming is a way to select good paths without having to reconstruct RGB images for every single possible path.

Dev: Right, Rosa? The big idea seems to be that they tackle the problem of candidate-wise RGB rollout being super costly and slow by sharing visual encoding across all the potential paths and jointly predicting their future representations at multiple horizons.

Taro: So, if I understand this right, they aren't doing a full world model reconstruction for every choice; instead, they use a latent predictor to map the history and each candidate action sequence to future features, and then a scorer uses those predictions to pick the best trajectory.

Rosa: Exactly. They use this latent predictor that takes the observation history and each candidate action sequence to predict future features across four horizons, H1 through H4. This means they're looking ahead in time for all of them at once without needing a full visual rollout for every option.

Dev: And the way they do that involves encoding the observation history and goal only once per state using a shared frozen DINOv3 encoder and a lightweight adapter, which is key to making it efficient. Then, for each candidate trajectory, they get action tokens and cumulative poses to initialize queries for predicting those future latents.

Taro: So the action-conditioned world model predicts these future latents jointly without needing autoregressive feedback when looking at the history and goal conditions. That sounds like a clever way to keep it fast while still capturing the necessary foresight.

Rosa: And after they get those predicted futures, they feed them into a learned scorer that combines those predictions with the current state and goal to predict how much the outcome will change relative to what’s currently happening with the incumbent trajectory.

Dev: That scorer predicts metrics like Delta ATE, Delta RPEtrans, Delta FDEi, and Ji based on these predicted evolutions compared to the incumbent path. The selection score is then built using those predicted outcomes along with some context from all the proposed trajectories to guide which one they actually choose or if they stick with the current one.

Taro: So, for someone just listening, this means instead of testing every potential route by running a full visual simulation for each, LiteNWM uses these joint predictions to score them efficiently. It’s about using foresight in the latent space to make trajectory selection faster and more reliable when things get tricky out in the wild.

Rosa: That's a good way to put it. So, we have this paper called LiteNWM: Efficient Latent World Models for Onboard Visual Navigation in the Wild, and what we've covered is how they use latent space prediction combined with relative scoring to manage candidate-wise RGB rollout costs.

Dev: And looking at the context of this paper, it seems like they’re focusing heavily on speed and efficiency gains over existing methods like NoMaD+NWM-XL. They show a significant speedup, reaching one hundred twenty-eight point zero zero times faster on an RTX five thousand ninety when compared to NoMaD+NWM-XL on that hardware <ref:2610.12368#pg2>.

Taro: That comparison is interesting because it shows the practical performance gain when you actually run these things on real hardware in challenging environments, like indoor and outdoor settings. The results show a jump in navigation success from about forty-three percent up to eighty-three percent relative to NoMaD.

Rosa: So, what this really suggests for people using this kind of system is that it can make the difference between failing a mission and succeeding in the field, especially when you're dealing with unseen environments where things don't go exactly as expected.

Dev: And they also showed that the frozen evaluator from their model can be transferred over to MBRA without needing to retrain it specifically for each proposer, which is a nice piece of engineering because it reduces the effort required for deployment on other systems.

Taro: The paper also highlights that they've shown cross-proposer transfer capabilities, meaning the system is more robust when dealing with different starting proposals in its proposal set. That’s important for real-world autonomy where you might not always get a perfect initial suggestion.

Rosa: So, to wrap up what we've heard about LiteNWM: Efficient Latent World Models for Onboard Visual Navigation in the Wild, it’s a system that uses action-conditioned multi-horizon latent prediction and incumbent-relative scoring to select trajectories from a frozen policy without having to reconstruct RGB images.

Dev: And looking at the title and what we've discussed, this work is really about making onboard visual navigation more efficient by cleverly restructuring the evaluation path around shared encoding and joint candidate-future prediction.

Taro: It points toward a future where models can make these foresight predictions very cheaply, which could open up possibilities beyond just basic navigation into more complex manipulation tasks.

Rosa: Right. So this paper shows that by focusing on how to predict the consequences of different actions in latent space, we can get much better performance and speed when deploying autonomous systems in the real world compared to traditional methods like NoMaD.

Conclusion: Rosa: So we're wrapping up on LiteNWM and I just want to make sure we've got the core idea straight for everyone listening right now, right?

Dev: Basically, this paper is about taking a really heavy way of planning paths—the one that has to reconstruct the whole scene for every single option—and making it much lighter by using latent representations instead.

Taro: It’s about selecting a path without doing all that extra visual work on every candidate trajectory.

Rosa: Right, so LiteNWM uses something called action-conditioned multi-horizon latent prediction and incumbent-relative scoring to pick the best route from a frozen policy.

Dev: That means they aren't rebuilding the RGB image for every possible move; they're predicting what will happen in a few steps using learned representations.

Taro: And that selection process uses those predictions to see how much better or worse a path is compared to the one you’re currently on.

Rosa: It’s about making trajectory selection faster and more reliable when you're actually out there navigating something unpredictable in the real world.

Dev: The authors claim significant speedups, like one hundred twenty-eight times faster on certain hardware, but they also show real-world success in tough conditions compared to older methods.

Taro: It suggests that this approach of using latent foresight could open up new ways to handle complex tasks beyond just simple navigation down the road.

Rosa: That’s what we’ve been discussing so far about LiteNWM, and next, we're going to look at how they actually trained this whole system.

Linkai Liu, Yuntian Zhang, Zhenshan Bing, Chen Chen, Lingjuan Lyu, Shangguang Wang, Mengwei Xu, Dongqi Cai

Nanjing University 2

cs.RO

Submitted: 2026-10-08

Updated: 2026-10-08

The gist: The gist The LiteNWM model selects trajectories from a frozen policy without RGB reconstruction by combining action-conditioned multi-horizon latent prediction with incumbent-relative scoring How it

Key concepts

Latent Predictor
This component maps the robot's observation history and each candidate action sequence to future visual features. It uses a frozen DINOv3 encoder and action tokens to generate queries for predicting latent representations at four different future time steps (H1 through H4). These predictions are refined by integrating both action and history conditions without using autoregressive feedback.
Future-Aware Trajectory Scoring
A learned scorer takes the predicted future states from the latent predictor, along with the current state and goal. It uses these predictions to calculate metrics like changes in ATE (Average Time to Error) and RPEtrans (Relative Perceptual Error). This allows it to score candidate trajectories based on how well they are expected to perform relative to the current best path.
Incumbent-Relative Scoring
The scoring mechanism compares a proposed trajectory against the 'incumbent' (the currently executing or best known trajectory). It predicts outcome changes by comparing the predicted evolution of a candidate path against the incumbent's evolution. This comparison guides selection by focusing on how much better or worse a candidate is expected to perform compared to what is already known.
Action-Conditioned Multi-Horizon Prediction
Instead of generating full RGB images for every potential move, LiteNWM predicts future visual latents at multiple horizons (H1 to H4) conditioned on the current action and past observations. This allows the model to efficiently encode the necessary visual information required for decision-making without needing a full, slow RGB rollout.

Terminology

Summary

The gist The LiteNWM model selects trajectories from a frozen policy without RGB reconstruction by combining action-conditioned multi-horizon latent prediction with incumbent-relative scoring

How it works

LiteNWM is designed to address the bottleneck of candidate-wise RGB rollout by sharing visual encoding across candidates and jointly predicting their action-conditioned future representations at multiple horizons The core components of LiteNWM include a latent predictor that maps observation history and each candidate action sequence to future features, and a learned scorer that uses these predictions to select trajectories

LiteNWM operates by first encoding the observation history and goal once per state using a shared frozen DINOv3 encoder and a lightweight adapter Then, for each candidate trajectory τi, the action encoder returns action tokens and cumulative poses which are used in pose-conditioned warping to initialize queries for predicting future latents across horizons H1 through H4 These queries are refined through six factorized blocks that integrate action and history conditions to predict the future latents jointly without autoregressive feedback

Future-Aware Trajectory Scoring

The predicted futures are then fed into a learned scorer which combines these predictions with the current state and goal to predict outcome changes from the incumbent The scorer predicts metrics such as ∆ATE, ∆RPEtrans, ∆FDEi, and Ji based on the predicted evolutions relative to the incumbent trajectory The selection score is then defined using these predicted outcomes and proposal-set context to guide trajectory selection

Training, Trajectory Selection, and Execution

The training process involves sequentially training three-member predictor and scorer ensembles while keeping NoMaD and DINOv3 frozen The predictor losses comprise latent regression, cosine, action-sensitivity, temporal-consistency, and uncertainty terms The scorer losses include heteroscedastic regression and dualimprovement/harm classification alongside pairwise/listwise ranking objectives Trajectory selection is governed by an eligibility test that defines a learned acceptance gate based on criteria such as ui,ATE < −mATE d and p dual i ≥ θ dual d

Performance and Deployment

LiteNWM demonstrates significant performance gains in offline evaluations on RECON, SCAND, and SACSoN datasets On matched initial NoMaD proposal pools across the three domains, LiteNWM reduces macro ATE by 17.56% over NWM-XL In real-robot experiments in unseen indoor and outdoor environments, LiteNWM improves navigation success from 43.3% to 83.3% relative to NoMaD The model achieves end-to-end speedups of 128.00× on an RTX 5090 compared to NoMaD+NWM-XL Furthermore, the frozen evaluator transfers to MBRA without proposer-specific retraining, reducing MBRA’s macro-averaged trajectory error by 16.2% In real-world deployment on a Unitree Go2 EDU in unseen environments, LiteNWM improves navigation success from 43.3% to 83.3% over NoMaD

Conclusion and Future Work

LiteNWM successfully combines action-conditioned multi-horizon latent prediction with incumbent-relative scoring to select trajectories from a frozen policy without RGB reconstruction Future work aims to extend action-conditioned latent foresight beyond navigation to manipulation tasks, exploring how gradients of a smooth incumbent-relative cost could pass through the scorer, predictor, and candidate actions for training a differentiable generator The system is designed to continuously refresh the visual history and predict the H1–H4 latent consequences of proposed trajectories using a receding-horizon procedure The deployment on physical robots shows that LiteNWM achieves more reliable goal reaching than NoMaD across all tested settings The system's efficiency is quantified by profiling showing that candidate-wise RGB rollout accounts for 84.8% and 97.6% of the captured NWM-S and NWM-XL latency This dominant cost is mitigated by encoding the history and goal once per state, jointly predicting candidate futures across H1–H4 without autoregressive feedback<ref:2610.

Improvements for AI systems

  1. textbfCandidate-wise Latent Prediction for Trajectory Selection: Enables Future-Aware Planning without RGB Reconstruction. The system can select trajectories based on jointly predict[ing] their action-conditioned future representations at multiple horizons and uses a learned scorer uses these predictions to select trajectories, which reduces the costly candidate-conditioned RGB rollout by leveraging shared visual encoding.

  2. textbfRelative Outcome Evaluation with Proposal Context: Guides Trajectory Selection Using Predicted Consequences. The system employs a scorer that compares predicted evolutions relative to the incumbent, explicitly predicting outcomes like ∆ATE, ∆RPEtrans, ∆FDE, [and] ∆J to guide selection based on learned criteria such as dual improvement/harm classification.

  3. textbfCross-Policy Transfer for Robust Deployment: Allows Frozen Evaluators to Adapt to New Navigation Policies. LiteNWM can be deployed with a frozen evaluator that transfers to other navigation policies like MBRA without proposer-specific retraining, demonstrating the system's capability for deployment on different underlying navigation models.

  4. textbfReal-World Reliability in Unseen Environments: Achieves High Success Rates on Physical Robots. The system improves navigation success from 43.3% to 83.3% in real-robot experiments, showing it can perform closed-loop navigation in previously unseen indoor and outdoor layouts without scene-specific training or online model updates.

  5. textbf Enhanced Computational Efficiency: Significantly Reduces Inference Latency for Planning Decisions. By replacing candidate-wise RGB rollout with shared encoding and joint candidate-future prediction, the system achieves substantial speedups, such as a 128.00× end-to-end speedup on an RTX 5090 compared to NoMaD+NWM-XL.

Sources

Related papers