Can MLLMs Find Their Way in a City? Exploring Emergent Navigation from Web-Scale Knowledge

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Hi, I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Can MLLMs Find Their Way in a City? Exploring Emergent Navigation from Web-Scale Knowledge".

Jane: Here's the gist of the paper. Leveraging multimodal large language models (MLLMs) to develop embodied agents offers significant promise for addressing complex real-world tasks, but current evaluation benchmarks rarely probe the nuanced, knowledge-intensive reasoning essential for practical scenarios CityNav Benchmark and Task Definition The paper introduces Sparsely Grounded Visual Navigation as a task explicitly designed to evaluate the sequential decision-making abilities of MLLMs in challenging, knowledge-intensive real-world environments This task is operationalized with CityNav, a comprehensive benchmark constructed encompassing four diverse global cities to assess raw MLLM-driven agents in city navigation Agents must rely solely on visual inputs and internal multimodal reasoning to sequentially navigate 50+ decision points without additional environmental annotations or specialized architectural modifications Verbalization of Path (VoP) Mechanism To address the performance gap, the authors propose Verbalization of Path (VoP), a mechanism designed to explicitly ground the agent’s internal reasoning by probing city-scale cognitive maps from the MLLM This is achieved by augmenting the agent’s prompt-as-policy framework with three distinct phrases: "Write the exact location of the destination", "Write the current estimated exact location", and "Write the walking directions from the current position to the destination" This mechanism substantially enhances navigation success, as demonstrated by AgentNav achieving a 92% success rate on the New York split AgentNav Memory Components For extensive runs averaging over 50+ decisions per trajectory, the paper decomposes memory management into three core components to enhance efficiency within a POMDP framework This mechanism transforms the partially observable process into a Markovian one within an augmented state space s˜t = (vt, mt) which eliminates the need for full episodic memory The other components include Decision History, which maintains a structured record of the sequence of actions chosen by the agent at each intersection Ht = opartion sequence of actions chosen by the agent at each intersection during the trajectory and Previous Visit, which provides awareness of past interactions at specific intersections by encoding a visit count nv for each node v Evaluation Results and Limitations Empirical evaluation across six strong MLLMs shows that AgentNav consistently and substantially improves performance over base MLLMs, indicating the strength of the VoP method However, state-of-the-art reasoning techniques like GEPA and reflection baselines fail to attain competitive results on this long-range embodied navigation task suite Furthermore, the evaluation protocol depends on Google Street View imagery and its underlying graph topology, which can exhibit structural inconsistencies such as missing reverse edges Ablation Analysis Ablations show that incrementally integrating memory components highlights their individual impacts: incorporating Markovian Memory increases success to 23%, and further addition of Decision History boosts performance to 29% Finally, integrating Previous Visit yields an additional enhancement, reaching a success rate of 35% The stepwise analysis shows that Partial VoP enhances success to 66%, and complete verbalization reaches the highest performance with a 92% success rate AgentNav Prompt Structure The AgentNav prompt requires the agent to return a JSON object strictly matching a schema containing analysis, decision, and memory fields This structure ensures that the agent's internal reasoning and navigation strategy are explicitly grounded in external world knowledge through the structured prompting The proposed Verbalization of Path mechanism, complemented by strategic memory components, effectively coaxes the intrinsic world knowledge from MLLMs, resulting in significant performance improvements over existing reasoning methods This work highlights current limitations of MLLMs for embodied sequential decision-making tasks and emphasizes the need for continued research in robust, adaptive reasoning frameworks.

Title and authors: Tom: Okay, let's get into the title of this paper: "Can MLLMs Find Their Way in a City? Exploring Emergent Navigation from Web-Scale Knowledge". It’s pretty direct, asking if these models can actually navigate cities on their own when they have been trained on web-scale data.

Jane: That question gets to the heart of it, doesn't it? Are we just training models that can recognize pictures of streets, or are we seeing something more emergent happen—something like actual navigation capability—when they encounter a new city?

Lu: The authors are Utkarsh Mishra and Narendra Ahuja from the University of Illinois Urbana-Champaign, with Nebojsa Jojic from Texas A andM University at Microsoft Research. Their background in deep learning and vision models is what sets up their approach to this kind of embodied navigation research.

Meng: I’m looking at their affiliations, and it shows a strong collaboration between academic AI labs and Microsoft Research, which usually means they’re pushing on the cutting edge of how these large models can actually be applied in practical settings.

Lalam: Lalam sees this paper as an exploration into the limits of what pretraining can teach a model about spatial reasoning when that knowledge is applied to a completely novel navigation environment without any specific training for that task.

Tom: So, the core idea isn't just pattern matching; it’s testing if the web-scale knowledge they absorbed allows for this kind of spatial reasoning to emerge in an embodied agent operating in a city. It’s about generalization.

Jane: Exactly. They are checking if the broad knowledge gained from interleaved image-text corpora enables these foundation models to generalize their abilities across diverse vision-language tasks, which is what they claim leads to sophisticated agents eighteen twenty-two fifty-eight <ref:2512.15933#pg2,across diverse vision-language tasks>.

Lu: They're leveraging that emergent behavior from pretraining to see if it can translate into robust zero-shot and few-shot generalization for these kinds of agents. It’s about testing if the knowledge they already have is sufficient for the task at hand.

Meng: From an engineering standpoint, I wonder how much of that generalized knowledge is actually useful versus just noise when you’re trying to get a model to make a specific sequence of fifty decisions in a city <ref:2512.15933#pg3>.

Lalam: The paper suggests that this emergent behavior has already enabled foundation models to exhibit robust general reasoning and instruction-following abilities two eight forty forty-one sixty-one, which is what makes them even capable of tackling these kinds of complex navigation problems <ref:2512.15933#pg2,enabled foundation models to exhibit robust general reasoning and>.

Tom: So they are starting from the premise that pretraining gives us a strong foundation for this kind of agent behavior. It’s not about teaching it every corner; it’s about seeing if it already knows enough to figure out the route.

Jane: And what they're demonstrating is that this emergent behavior can be leveraged, but only with some targeted interventions to make the navigation succeed in a specific, challenging setting like city navigation in the wild.

The paper's summary: Tom: So we’ve seen how they set up this task—Sparsely Grounded Visual Navigation—which is essentially testing if agents can navigate fifty plus decision points sequentially without any extra environmental annotations or specialized architectural changes <ref:2512.15933#pg3>.

Jane: That’s a tough setup, and the paper summarizes that the goal is to see if MLLMs can perform spatial reasoning and strategic planning to execute routes to their destinations based only on visual input.

Lu: The summary highlights that current state-of-the-art MLLMs, including GEPA, chain-of-thought, and reflection techniques, and even competitive baselines like PReP significantly underperform in this challenging setting.

Meng: That tells me that the standard ways we're trying to guide these models with reasoning are not sufficient for this level of real-world navigation complexity. They’re not cutting it when you need sequential planning in an open city.

Lalam: To address this gap, they propose Verbalization of Path, or VoP, as the key mechanism. This method explicitly grounds the agent’s internal reasoning by probing city-scale cognitive maps—things like key landmarks and directions toward the destination—from the MLLM itself.

Tom: So VoP is basically a way to force the model to use its internalized city map when it’s facing an open navigation problem. It’s not just relying on what it vaguely remembers; it's actively asking for specific pieces of information from itself.

Jane: And they show that this verbalization substantially enhances navigation success, citing a ninety-two percent success rate on the New York split as a key piece of evidence for its effectiveness.

Lu: The summary also points to the fact that their work involves extensive evaluations across four diverse global cities, like New York and others, to make sure it’s not just working in one specific map.

Meng: Testing across multiple cities adds a layer of complexity because city layouts are so varied; if it only works on one topology, it’s probably not generalizable. So the multi-city aspect is important for the results they claim.

Lalam: And they mention that their work involves fine-tuning models on specific outdoor VLN data sets, like Real Manhattan and M2S data, to prepare them for this kind of task before testing it in a real multi-city scenario.

Tom: So the summary boils down to: standard MLLMs struggle with long-range, unannotated city navigation, and they propose using VoP to ground their reasoning in city maps for better performance.

Jane: And that’s the central message for listeners who aren't reading the paper: current models are good at general knowledge but need a specific tool like VoP to actually use that knowledge effectively in complex physical tasks.

The paper's improvements: Tom: Now let’s talk about the actual proposed improvements they suggest for this research, because those changes show us exactly where they think the next steps should be taken.

Jane: They focus on making VoP even better by adding more structured prompts to really squeeze every bit of information out of the model's internal map. They are proposing three distinct phrases: "Write the exact location of the destination," "Write the current estimated exact location," and "Write the walking directions from the current position to the destination."

Lu: That’s a specific way they structure it, and it’s designed to elicit those city-scale cognitive maps more effectively. It moves beyond just one type of probing and gives the model different kinds of directional guidance.

Meng: From my perspective on deployment, structuring the output into a JSON object with analysis, decision, and memory fields is a good thing because it makes it predictable and verifiable for engineering purposes. Predictability is key when you’re trying to put something into production.

Lalam: That structured prompt structure ensures the agent's internal reasoning and navigation strategy are explicitly tied to external world knowledge in a very specific, parsable way, which helps with verification as they say in their description of the AgentNav prompt structure.

Tom: I like that for grounding. But what about those memory components they talk about? They’re proposing a multi-component system to manage state for long runs of fifty plus decisions <ref:2512.15933#pg3>.

Jane: They are decomposing memory into three core parts: Markovian Memory, Decision History, and Previous Visit Tracking, with the goal of reducing the need for full episodic memory. This transforms the partially observable process into a Markovian one within an augmented state space s˜t equals vt and mt.

Lu: The other components they suggest are Decision History, which maintains a structured record of every action chosen by the agent at each intersection during its trajectory, and Previous Visit, which tracks past interactions at specific intersections by encoding a visit count nv for each node v.

Meng: So they’re trying to achieve this reduction in memory overhead by tracking just the essential sequence and past node visits rather than storing every single state detail for every step. That sounds like a necessary move if we want to run this on hardware that isn't infinitely powerful.

Lalam: The ablation analysis shows that incrementally adding these pieces matters: Markovian Memory alone increases success to twenty-three percent, and adding Decision History boosts it up to twenty-nine percent, while integrating Previous Visit yields an additional enhancement up to thirty-five percent.

Tom: So the stepwise breakdown is really telling. It proves that each memory component provides tangible value, but you need them all working together for the full effect, which leads to that ninety-two percent success rate with complete verbalization.

Jane: And they also have a suggestion for making the navigation even smoother: incorporating a dynamic temperature parameter T that anneals with increasing straight-line distance from an initial random exploration toward a more directional selection, which is supposed to mitigate loops.

Conclusion: Tom: Alright, let’s wrap this up on "Can MLLMs Find Their Way in a City? Exploring Emergent Navigation from Web-Scale Knowledge". The main conclusion is that the Verbalization of Path mechanism combined with strategic memory components effectively coaxes the intrinsic world knowledge from MLLMs for significant performance gains over existing reasoning methods.

Jane: So, it’s not just about one trick; it’s a combination of explicit prompting, like VoP, and smart memory management to get those agents to perform well in these complex tasks. It shows that leveraging existing model capabilities requires a targeted scaffolding approach.

Lu: The implication is that we need to continue researching more robust and adaptive reasoning frameworks because relying on raw emergent behavior alone isn't enough for real-world embodied sequential decision-making, even with large models.

Meng: For deployment, the caveat remains that the evaluation protocol depends on Google Street View imagery and its underlying graph topology, which they flag as potentially having structural inconsistencies like missing reverse edges. That means we can’t just deploy this without ensuring our map data is sound.

Lalam: Exactly. They also stress that the limitation is that their success rate isn't guaranteed because the environment itself might be flawed in ways that weren't accounted for, so it highlights a need for more resilient systems.

Tom: So, to finish up on this paper: "Can MLLMs Find Their Way in a City? Exploring Emergent Navigation from Web-Scale Knowledge", we see that VoP and the memory components are powerful tools to ground model reasoning and achieve high success rates.

Jane: It’s a win for showing how we can guide these large models to perform real-world embodied tasks by explicitly using their existing knowledge in a structured way. A lot of great thinking happening here.

Lu: And the next step is definitely building on this work to create systems that are more resilient when the underlying world data isn't perfect, focusing on those structural inconsistencies they pointed out.

Meng: I’d like to see research that focuses less on just hitting a high percentage and more on how these agents handle unexpected failures during long-range navigation runs. That’s where the real engineering challenges are hiding.

Lalam: It really shows that the path forward involves not just better prompting, but creating entire frameworks around these agents that are adaptive to the specific limitations of their operational environment.