GROW squared: Grounding Which and Where for Robot Tool Use

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "GROW squared: Grounding Which and Where for Robot Tool Use".

Dev: The gist: GROW2 introduces a novel approach for open-world affordance grounding in robot tool use by decomposing the problem into semantic selection and geometric grounding using object parts,

Rosa: First, who's behind it and why it matters.

Title and authors: Rosa: So we're looking at a paper called "GROW squared: Grounding Which and Where for Robot Tool Use." It sounds like they're trying to figure out how robots can use tools they haven't explicitly been trained on, which is a big deal for open-world robotics.

Dev: Yeah, it tackles the challenge of open-world affordance grounding, which is essentially figuring out which object to use as a tool and exactly where on that object the robot should interact with it. It tries to do this without relying on huge datasets for every single possible tool and action.

Taro: I'm curious about what makes it different from existing methods because a lot of them assume you already know the exact tool beforehand, which is not realistic in a cluttered space.

Rosa: Exactly, that’s where they introduce this two-level decomposition using object parts as the main abstraction to split the grounding process. It separates picking the right tool and figuring out where to grasp it into two distinct steps.

Dev: So at the semantic level, they use vision-language models or VLMs to do the reasoning part, which means interpreting a natural language instruction like "cut the cake."

Taro: And then at the geometric level, they use vision foundation models to take those selected parts and ground them into precise three dimensional affordance regions from just one RGB-D image. That seems like a powerful way to handle ambiguity.

The paper's summary: Rosa: The core idea here is that by splitting the problem this way, they can leverage different foundation models for generalization instead of needing one massive model trained on every possible tool and action.

Dev: They use the VLM to select a suitable object as the tool and then identify task-relevant parts on both that tool and the target object. That handles the "Which" part of the problem, figuring out what to use and where to look.

Taro: And then they take those selected parts, say a screwdriver handle and a piece of wood, and use vision foundation models to ground them into three dee affordance regions from a single view <ref:2606.30632#pg1>. That addresses the "Where" part.

Rosa: They state that this decomposition eliminates the need for large-scale affordance annotations and lets the system generalize to open-category objects, which is what they call GROW2, or GROW squared.

Dev: They show that this approach allows them to produce complete and function-faithful three dee affordance regions from just a single view of an observation, which is something multi-view methods struggle with because they are inefficient for real deployment <ref:2606.30632#pg1>.

Taro: So the implication is they can potentially get a complete understanding of how to use any object as a tool based on its parts and the task instruction, without needing massive datasets for that specific interaction.

The paper's improvements: Rosa: One major improvement is that they move away from needing substantial three dee training data for traditional methods and instead rely on this part representation to decompose the grounding process <ref:2606.30632#pg1>.

Dev: They also address the fact that most prior work assumes you have a complete three dee observation, which isn't realistic, so GROW2 tackles that by using a single-view RGB-D image as input <ref:2606.30632#pg1>.

Taro: They handle occlusion challenges by combining 2D part grounding with three dee reconstruction <ref:2606.30632#pg1>. They reconstruct a high fidelity mesh, render it from multiple viewpoints, segment the parts in each view using SAM3, and then fuse those masks into complete three dee affordance regions <ref:2606.30632#pg1>.

Rosa: To make that geometric grounding more robust when the VLM might struggle with text queries, they use a vision-language model as a fallback to translate complex text queries into bounding boxes when SAM3 might not understand them.

Dev: And for the final step in finding the actual region, after getting those 2D segmentations, they back-project them onto the reconstructed mesh and use DBSCAN to cluster the points and keep only the largest cluster as their predicted affordance region <ref:2606.30632#pg1>.

Conclusion: Rosa: So, to wrap up on "GROW squared: Grounding Which and Where for Robot Tool Use," this paper presents a method that grounds affordances by decomposing the problem into semantic selection and geometric grounding using object parts.

Dev: It shows that by using VLMs for semantic selection and vision foundation models for geometric grounding, they can create complete three dee affordance regions from a single view observation <ref:2606.30632#pg1>.

Taro: The real world experiments on a Franka arm show that this method works reliably even with noisy real-world depth observations because it computes the affordance regions from the reconstructed mesh through multi-view rendering.

Rosa: The limitation they state is that failures can still happen across all fifty real-world trials, and tool and part selection accounts for the largest share of those failures. Plus, processing an entire scene takes about sixteen point six seconds, which points to a need to distill this into something much lighter.

Dev: That latency from querying VLMs and running inference on foundation models makes dynamic manipulation challenging right now. But overall, it's a zero-shot method for grounding affordances that can effectively support downstream robot tool use.