AgenticNav: Zero-Shot Vision-and-Language Navigation as a Tool-Calling Harness

summary

The gist

Zero-shot vision-and-language navigation in continuous environments (VLN-CE) has recently become feasible with large vision-language models (VLMs), but existing methods suffer from limitations in

In short

AgenticNav rethinks zero-shot vision-and-language navigation by treating it as an agentic tool-calling process. It exposes action, depth, and memory as callable tools for a vision model. This harness allows the VLM to directly select targets, request metric depth for safety checks, and selectively recall past experiences from a compact map image. The method achieves state-of-the-art results by improving how the model interacts with its environment.

Key concepts

AgenticNav: A lightweight harness that reformulates zero-shot navigation as an agentic tool-calling process. It exposes core navigation capabilities like action, depth, and memory as specific tools that the vision-language model can call upon to interact with the environment.
Waypoint Predictor Limitation: Existing methods rely on a learned waypoint predictor, which limits the robot's movement choices to a small set of candidates. This approach fails to utilize metric depth information effectively for spatial reasoning and restricts the VLM's freedom of selection in novel scenes.
On-demand Pixel-Depth Tool: This tool allows the VLM to request precise metric depth measurements at specific image locations before taking an action. This enables targeted spatial comparisons, such as verifying if a path is open or checking the reachability of a candidate point, enhancing spatial reasoning.
Selective Memory-Recall Tool: This mechanism manages navigation history by creating a compact map image summarizing the trajectory and providing a recall tool for selective revisiting. This prevents overwhelming the VLM's context with irrelevant history while allowing it to retrieve specific past visual observations.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Federated Learning — explained in 16 episodes
Retrieval-Augmented Generation (RAG) — explained in 16 episodes
Interpretability — explained in 15 episodes
Sample Complexity — explained in 15 episodes
Generalization — explained in 13 episodes

This episode discusses

The paper

AgenticNav: Zero-Shot Vision-and-Language Navigation as a Tool-Calling Harness · Read on arXiv

Shanghai Jiao Tong University

Zero-shot vision-and-language navigation in continuous environments (VLN-CE) has recently become feasible with large vision-language models (VLMs). Existing methods typically rely on learned waypoint predictors to propose navigable actions. This limits the model's action space and fails to leverage depth inputs effectively. Moreover, memory is commonly handled by accumulating long textual or visual histories, which overwhelms the prompt context. In this paper, we rethink zero-shot VLN-CE as an agentic interface between the VLM and the environment, and present AgenticNav, a lightweight navigation harness that exposes action, depth, and memory as callable tools. Instead of choosing from predicted waypoints, the action tool allows the VLM to directly select a target pixel in RGB observations, converting it into executable motion. Depth is exposed through an on-demand pixel-depth tool, enabling the VLM to request precise metric distances only where they matter. For memory, AgenticNav uses an agentic memory architecture combining reasoning text history and a compact map image, paired with a recall tool that allows the VLM to selectively revisit past visual observations. On R2R-CE, AgenticNav achieves state-of-the-art (SOTA) zero-shot performance under same VLM comparison, reaching 76% success rate and 66.50% SPL. Ablation experiments validate the effectiveness of our tool and memory designs, while backbone evaluations demonstrate more consistent navigation gains as VLMs improve, highlighting the harness's potential to benefit from future VLM advances. Real-world experiments on two distinct robotic platforms further demonstrate its zero-shot generalization and adaptability.

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "AgenticNav: Zero-Shot Vision-and-Language Navigation as a Tool-Calling Harness".

Dev: Zero-shot vision-and-language navigation in continuous environments (VLN-CE) has recently become feasible with large vision-language models (VLMs), but existing methods suffer from limitations in action space, depth utilization, and memory management.

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: Moving onto the title and authors, we’re looking at "AgenticNav: Zero-Shot Vision-and-Language Navigation as a Tool-Calling Harness." The authors are Yijian Li, Changze Li, Hantian Shi, Jiaying Luo, and Jiyuan Cai.

Dev: I see the focus there is on making zero-shot navigation feasible by changing *how* the VLM interacts with the environment rather than just bigger models alone.

Taro: It seems like they are tackling a fundamental problem: how to give a VLM the ability to reason about continuous space without needing massive amounts of task-specific training data for every single scenario.

Rosa: Exactly, Taro; the authors frame it as rethinking zero-shot VLN-CE by treating navigation as an agentic interface that exposes action, depth, and memory as callable tools. This means we’re not just teaching a policy; we’re giving the model a set of explicit actions it can request from its surroundings.

Dev: That sounds like they're trying to solve the problem of poor action space limitations by letting the VLM directly select a target pixel in RGB observations, which is much broader than choosing from a small set of waypoints.

Taro: I wonder how this direct selection capability plays out when the instruction is vague, and what kind of feedback we get if that selection leads to an impossible or unsafe situation.

Rosa: The paper details the "Action Tool" specifically, which takes a selected pixel and returns either an execution command or feedback after it back-projects the target into a three dee point and performs a geometric safety check to ensure no body-height point falls inside the swept corridor <ref:2606.10577#pg1>.

Dev: That deterministic safety checking is something I’m really interested in; having that hard constraint baked into the tool helps manage some of those failure modes we worry about in control systems.

The paper's summary: Rosa: Now, let’s look at the actual summary of "AgenticNav: Zero-Shot Vision-and-Language Navigation as a Tool-Calling Harness." They explain that they reformulate zero-shot VLN-CE as a tool-calling interface that exposes depth, memory, and action grounding directly to the VLM.

Dev: It seems like the core idea is to replace learned predictors with these three explicit tools: an action tool, a depth tool, and a selective memory recall tool.

Taro: So the summary highlights that they are addressing three main bottlenecks in existing methods: restricted action spaces from waypoints, ineffective depth utilization because it wasn't explicitly exposed to the VLM for spatial reasoning, and context overload from long histories.

Rosa: That’s right; the paper points out that waypoint predictors restrict the action space by only choosing among a small set of candidates, and they also noted that waypoint-based interfaces take depth inputs as ground truth but don't explicitly expose them to the VLM for spatial reasoning.

Dev: And then there’s memory, which they say is often handled by accumulating long textual or visual histories with substantial irrelevant context, or by retrieving cross-episode experiences, which weakens the zero-shot setting.

Taro: I see how that selective memory tool tries to fix the context accumulation issue by combining a compact trajectory image map with a recall tool that lets the VLM selectively revisit past observations without overwhelming its prompt.

The paper's improvements: Rosa: Focusing on what they actually improved, AgenticNav introduces several specific enhancements. They suggest that replacing the Action Tool with a waypoint predictor reduces performance metrics like SR and SPL, which confirms their point about needing to choose visual targets directly instead of being restricted to learned waypoints.

Dev: I’m paying attention to the depth tool because they show that requesting metric depth at precise image locations allows the VLM to leverage that information more effectively than traditional waypoint predictors or direct depth-image inputs for spatial reasoning.

Taro: The paper also points out the memory mechanism as a major improvement; combining a compact trajectory image map with selective visual recall actually improves long-horizon decision-making while avoiding the accumulation of long historical-frame contexts.

Rosa: And they highlight that by designing this agentic harness, they manage to demonstrate state-of-the-art zero-shot performance on the R2R-CE benchmark under fair VLM backbone comparisons, which is a big win for their methodology.

Dev: It's interesting that the ablation study confirms this; removing the Depth Tool drops SR significantly, showing how crucial that spatial information is for their navigation success.

Conclusion: Rosa: To wrap up our discussion on "AgenticNav: Zero-Shot Vision-and-Language Navigation as a Tool-Calling Harness," the paper successfully rethinks zero-shot VLN-CE by treating navigation as tool calling, moving away from reliance on trained waypoint predictors.

Dev: It’s clear that this harness removes the dependency on those learned predictors and gives the VLM better access to spatial and contextual information through dedicated tools for action, depth, and memory.

Taro: I think the most important implication is that they show that the design of these action, perception, and memory harnesses is as important as the choice of the VLM itself for foundation-model navigation.

Rosa: Precisely; this paper demonstrates that when you design a robust interface like AgenticNav, it can unlock performance gains even with models like GPT-five point five or Gemini-two point five-Pro, showing strong sim-to-real generalization as well.

Dev: From an engineering standpoint, the deterministic safety checking integrated into the action tool is a huge plus for reliability in continuous environments where things can get messy quickly.

Taro: I just want to reiterate that their findings on the R2R-CE benchmark under fair VLM comparisons really show how much more robust these agentic interfaces are compared to prior methods.

Rosa: That’s right; AgenticNav provides a concrete way forward for getting powerful foundation models into truly continuous and reliable navigation tasks.

Dev: We're looking forward to seeing how this harness integrates into real-world hardware and what kind of latency we can expect when deploying these tool calls in production.

More episodes

← Home