G2-Nav: Grounded and Guarded Vision-Language Costmaps for Robot Social Navigation

arXiv:2607.16956 · cs.RO · Submitted 2026-07-18 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: I'm Rosa, and with me are Dev and Taro, guest researcher.

Dev: Today's paper: "G2-Nav: Grounded and Guarded Vision-Language Costmaps for Robot Social Navigation".

Rosa: Social navigation requires robots to reason and respond in complex real-world environments, and G2-Nav addresses this by grounding abstract social reasoning into reliable,

Dev: First, who's behind it and why it matters.

Title and authors: Rosa: Let's start by looking at the title and the authors for this paper, "G2-Nav: Grounded and Guarded Vision-Language Costmaps for Robot Social Navigation." The name itself really tells you what they're trying to achieve here.

Dev: I see it’s a framework that explicitly grounds abstract social reasoning into costmaps, which is a key distinction from just using a VLM to give direct commands, Rosa. It suggests they are building something more structured than just asking the AI for a plan every time.

Taro: The authors are from NTU Singapore, and this paper seems to be addressing that gap where existing end-to-end VLM approaches create unpredictable black boxes, so grounding it in a costmap offers reliability.

Rosa: Precisely; they are aiming to bridge the gap between high-level semantic reasoning from VLMs and reliable robot behaviors by creating an interpretable interface. This means we can actually see *why* the robot chose a certain path, not just that it did.

Dev: That interpretability is vital for debugging; if something goes wrong in a real deployment, we need to trace back through that costmap formulation to see where the decision went astray.

Taro: The focus on grounding social context using open-set perception suggests they are building a system that can handle novel situations by recognizing objects and then interpreting their social meaning.

Rosa: It’s about taking the semantic understanding from the VLM—like knowing who is a potential guide or where traversable ground is—and turning that into something physical for the robot's planning engine to use.

Dev: So, instead of a pure instruction-following heatmap, they are using this costmap as a mathematically sound interface for those abstract social concepts, which sounds like it addresses some of the limitations of previous work.

Taro: The implication here is that we aren't just building another navigation system; we’re developing a way to translate complex human social understanding into robot action space efficiently.

Rosa: That’s the big picture—taking what the VLM understands about people and spaces and making it actionable for a physical machine in a way that prioritizes safety.

Dev: And I'm interested in how they structure this translation process, because if it’s too slow or inaccurate, all that social reasoning is useless when you're operating at high frequency.

Taro: They tackle this by breaking the problem down: first open-set perception to get the raw data, then VLM analysis for cues, and finally mapping those cues into the costmap structure.

Rosa: So they’re layering their approach: perception feeds reasoning, and reasoning feeds a structured map that dictates motion planning.

The paper's summary: Rosa: Now let's get into what the paper actually summarizes about G2-Nav; essentially, it outlines the framework of taking semantic reasoning from a Vision-Language Model and translating it into a vision-language costmap to serve as an interface for social navigation.

Dev: The core idea is that the VLM doesn't directly command movement but instead evaluates traversable regions and social agents based on open-set perception, which then maps that context onto this costmap.

Taro: So, the VLM is used to figure out where things are and what they mean socially—like identifying ground regions or scoring relevant objects based on danger or potential guidance.

Rosa: That social context is then put into a vision-language costmap, which mathematically combines standard navigation terms like goal attraction and static obstacles with these new social components.

Dev: I see how this contrasts with older methods that either just treat the occupancy grid as the costmap for walls or assign simple pre-defined costs to humans, ignoring more diverse social interactions.

Taro: The paper emphasizes that the unique capability of VLMs in social navigation is analyzing complex and unstructured real-world environments to identify and analyze interested agents, which is where they see an opportunity.

Rosa: They then apply upstream verification where the VLM checks if object depth and heading match what’s visually observed; if there’s a mismatch, they correct the depth or increase the social score.

Dev: That verification step sounds like a necessary filter to ensure that the abstract reasoning isn't based on faulty sensor data, which is something we have to worry about constantly in these systems.

Taro: It’s important because it shows how they handle uncertainty by using the VLM not just for prediction, but for cross-referencing perception against visual input.

Rosa: And they also introduce a specific costmap component called the Traversability Mask, which penalizes regions outside a binary mask derived from the VLM to enforce traversability rules.

Dev: So they’re combining obstacle avoidance with dynamic social constraints through this weighted combination formula, C = λ1Cgoal + λ2Cobs + λ3Cobj + Ctrav.

Taro: This formulation gives us a clear mathematical structure for how the robot weighs its immediate goal against the static environment and the dynamically changing social landscape.

Rosa: It really lays out a comprehensive pipeline: perception, semantic reasoning from the VLM, verification, costmap formulation, and finally using that map to generate control actions.

Dev: This sounds like a very thorough way to integrate high-level intelligence into low-level path planning without letting the system become completely opaque.

The paper's improvements: Rosa: Moving on to the specific improvements suggested by G2-Nav, it focuses heavily on incorporating reliability and robustness into this framework for real-world use.

Dev: The upstream verification mechanism is a major improvement because it uses the VLM to verify object depth and heading against visual input, correcting errors through depth re-registration or by increasing the social score if there's uncertainty.

Taro: That’s something I really like; it means that even if one part of the perception chain gets confused, the system has a built-in way to recover plausibility, which is essential when dealing with unpredictable human behavior.

Rosa: And then they have this high-frequency safety check called a Reflex Zone designed to catch latency failures; any unregistered LiDAR points entering that zone trigger an immediate high penalty in the costmap, forcing an urgent robot response.

Dev: That reflex zone addresses the potential for system lag causing dangerous situations; if we can define that zone well, it provides a fast way to inject emergency constraints directly into the planning process.

Taro: This safety layer is what makes this approach suitable for deployment in humancentric social environments because it adds a layer of real-time reactive safety on top of the planned navigation.

Rosa: And qualitatively, they show that this approach promotes "social compliance while preserving safety and efficiency in real-world navigation," suggesting a nice balance between the two goals.

Dev: I’m thinking about the limitations they mention; one point is that the VLM's performance dictates how good the social context is, meaning if the VLM misinterprets a cue, it translates into an error in our costmap.

Taro: That’s a fair limitation; it highlights that this framework's strength relies heavily on the accuracy of the social cue extraction and scoring performed by the VLM itself.

Rosa: So, while they solve the problem of abstract reasoning, we still have to ensure that their VLM isn't introducing new, subtle forms of error into our navigation decisions.

Dev: Exactly; it’s a trade-off between achieving sophisticated social awareness and maintaining the stringent reliability required for autonomous systems.

Conclusion: Rosa: Wrapping up this discussion on G2-Nav: essentially, this paper successfully grounds VLM semantic reasoning into reliable costmap-based robot behaviors through novel representation, upstream verification, and high-frequency safety checks.

Dev: The implication for us is that we have a way to handle complex social navigation in unstructured environments without relying solely on pre-defined costs or simple reactive avoidance schemes.

Taro: I think the real impact here is showing that we can use vision-language models not just for perception, but as a structured reasoning engine for planning in human-centric settings.

Rosa: It definitely sets a new direction for how we integrate large models into robot autonomy by focusing on creating interpretable and safe interfaces rather than just relying on end-to-end black boxes.

Dev: For engineering, it means we have concrete mechanisms—the costmap structure and the safety checks—that allow us to manage the latency and failure modes associated with social planning more explicitly.

Taro: The future work they mention points toward further refinement of this framework to handle even more complex, dynamic social scenarios where agents constantly change their behavior.

Rosa: So, G2-Nav offers a very solid foundation for building robots that can navigate the real world by understanding and responding to social dynamics in a safe manner.

Nanyang Technological University Singapore

cs.RO

Submitted: 2026-07-18

Updated: 2026-10-01

Comments: CoRL 2026

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 77/100

The gist: Social navigation requires robots to reason and respond in complex real-world environments, and G2-Nav addresses this by grounding abstract social reasoning into reliable, interpretable

Key concepts

Vision-Language Costmap
This is a mathematical representation of the robot's environment that combines standard navigation data (like obstacles) with social information derived from a VLM. It allows the robot to plan paths based on both physical constraints and social context, making abstract reasoning actionable for movement planning.
Upstream Verification
This safety mechanism uses the VLM to check if detected objects' depth and heading match visual observations. If there is a mismatch, it corrects the error by re-registering the depth or increasing the object's social score, ensuring that the robot trusts its perception of social agents.
High-Frequency Safety Check (Reflex Zone)
This is a rapid safety layer designed to prevent system delays. It monitors LiDAR points along a predicted path; if an unclassified point appears in this zone, it immediately adds a high penalty to the costmap, forcing the robot to react urgently and avoid potential hazards instantly.

Terminology

Summary

Social navigation requires robots to reason and respond in complex real-world environments, and G2-Nav addresses this by grounding abstract social reasoning into reliable, interpretable vision-language costmaps while incorporating safety checks for deployment.

How it works

The framework translates semantic reasoning from a Vision-Language Model (VLM) into a vision-language costmap to serve as a mathematically sound interface for abstract social context. Instead of asking the VLM for direct planning decisions, G2-Nav uses the VLM to evaluate traversable regions and social agents from open-set perception, mapping the resulting social context into this costmap.

The process involves several key steps:

  1. Open-set perception is performed using RAM++ for object recognition and GroundingDINO for 2D bounding box association, followed by DBSCAN clustering to identify objects in the point cloud.

  2. A pretrained VLM is then used to extract social cues, specifically performing Traversability Identification to determine ground regions and a Social Scoring mechanism to assign scores to relevant objects based on danger and potential guidance from social leaders.

  3. Upstream verification is applied where the VLM verifies if the object’s depth and heading match visual observations; if a mismatch occurs, depth re-registration is performed, or the object's social score is increased.

Costmap Formulation

The overall social context is formulated into a vision-language costmap, represented as a weighted combination of standard navigation terms and social context:

(1) C = λ1Cgoal + λ2Cobs + λ3Cobj + Ctrav

This costmap is defined over the planning horizon W = [−X, X] × [−Y, Y] and includes four components:

  1. Goal Attraction (Cgoal): Penalizes the Euclidean distance from the current location to the goal g.

  2. Static Obstacles (Cobs): This is the occupancy map projected from LiDAR point cloud P, inflated by a robot radius.

  3. Social Agent (Cobj): Models social interaction as a sum of elliptical Gaussians: Cobj = Σ sj · N (xj, Σ(vj)).

  4. Traversability Mask (Ctrav): Penalizes non-traversable regions outside the binary mask M derived from the VLM.

Safety and Robustness Mechanisms

To ensure real-world robustness, G2-Nav introduces two critical safety layers:

  1. Upstream Verification: This mechanism uses the VLM to verify object depth and heading against visual input, correcting errors through depth re-registration or by increasing the social score for uncertain motion.

  2. High-Frequency Safety Check (Reflex Zone): To guard against system latency, a dynamic reflex zone is defined along the robot’s predicted trajectory. LiDAR points falling within this zone that do not belong to a known object trigger an immediate high penalty injection into the costmap, forcing an urgent robot response.

Evaluation and Results

The framework was evaluated in two settings: recorded social navigation dataset SCAND and interactive deployment in a crowded campus environment. In the recorded setting, G2-Nav demonstrated superior performance compared to baselines like Dynamic Window Approach (DWA), Social Force (SF), PeopleAsPlanner (PAP), and CityWalker, achieving a lower L2 Distance and Maximum Average Orientation Error (MAOE). Specifically, G2-Nav outperformed CityWalker in both metrics. In the interactive setting, G2-Nav showed strong safety awareness without compromising efficiency, outperforming SF and PAP. Qualitative analysis confirmed that G2-Nav promoted social compliance while preserving safety and efficiency in real-world navigation.

Conclusion

G2-Nav successfully grounds VLM semantic reasoning into reliable costmap-based robot behaviors, promoting safe, efficient, and socially responsible navigation through its novel representation, upstream verification, and high-frequency safety checks. The framework is demonstrated to be robust for deployment in unstructured environments.

The gist: G2-Nav proposes a novel framework that grounds abstract social reasoning into reliable vision-language costmaps by using a VLM to map social context into a bounded cost space, augmented with upstream verification and a high-frequency safety check for safe real-world deployment.

The process involves several key steps:

  1. Open-set perception is performed using RAM++ for object recognition and GroundingDINO for 2D bounding box association, followed by DBSCAN clustering to identify objects in the point cloud.

Improvements for AI systems

Here are the specific improvements an AI system, based on the G2-Nav framework, can achieve:

  1. The AI system will be able to perform safe, efficient, and socially compliant autonomous navigation in unstructured real-world environments (e.g., crowded campus corridors).

  2. The system will move beyond simple obstacle avoidance by incorporating complex social reasoning into its planning process through a vision-language costmap. It can differentiate between ground regions, interested social agents (people), and irrelevant objects using a pretrained VLM.

  3. The AI will maintain robustness against noisy sensor data by implementing an Upstream Verification mechanism where the VLM checks the plausibility of object depth and heading against visual observations, correcting erroneous tracking clusters in real-time.

  4. The system will prevent latency-induced failures by incorporating a high-frequency Safety Reflex Zone check that triggers immediate high penalties in the costmap if unregistered LiDAR points enter the robot's predicted path, ensuring rapid emergency responses to sudden, unmodeled threats.

  5. The AI will generate trajectories that are socially aware and compliant, such as following human leaders or giving way to approaching groups based on semantic cues extracted by the VLM scoring system (e.g., assigning negative scores to humans providing implicit guidance).

  6. The system will operate as an embodiment-agnostic navigation framework by using a costmap that combines standard navigation terms (goal attraction, static obstacles) with dynamic social interaction models (elliptical Gaussians for social agents), allowing it to handle diverse interactions beyond simple repulsive forces.

  7. The system can be deployed in interactive, real-world settings where the VLM provides continuous context updates during every inference cycle, leading to a smoother and more context-aware planning process compared to methods relying on static or low-frequency sampling.

Sources

Related papers