VLMs Can Describe, But Not Measure: Object-Centric Scene Understanding for Robotic Manipulation

arXiv:2609.28184 · cs.RO, cs.CV · Submitted 2026-09-23 · Read on arXiv

cs.RO, cs.CV

Submitted: 2026-09-23

Updated: 2026-09-23

Code: https://github.com/Pointcept/Pointcept

Terminology

• Foundation Model — 5× in this paper · explained in 3 episodes
• Semantic Understanding — 5× in this paper · explained in 2 episodes
• Ablation — 4× in this paper · explained in 1 episode
• Foundation Models — 3× in this paper · explained in 7 episodes
• Grounding — 3× in this paper · explained in 3 episodes
• Instance Segmentation — 3× in this paper · explained in 3 episodes
• Task Planning — 3× in this paper · explained in 2 episodes
• Conference on Computer Vision and Pattern Recognition — 2× in this paper
• Conference on Robot Learning — 2× in this paper
• Fine-Tuning — 2× in this paper · explained in 5 episodes
• Generalization — 2× in this paper · explained in 13 episodes
• Transformer — 2× in this paper · explained in 2 episodes
• Accuracy — 1× in this paper · explained in 1 episode
• Affordance — 1× in this paper · explained in 1 episode
• Aggregation — 1× in this paper · explained in 1 episode
• Benchmark — 1× in this paper · explained in 2 episodes
• Consistency — 1× in this paper · explained in 3 episodes
• Point Clouds — 1× in this paper · explained in 2 episodes
• Structured Reasoning — 1× in this paper · explained in 3 episodes
• THE MEDIAN AND ROOT MEAN SQUARED ERROR — 1× in this paper
• Temporal Reasoning — 1× in this paper · explained in 2 episodes
• Vision-language models — 1× in this paper · explained in 2 episodes

Sources

• OpenAI GPT-5 System Card
• Describe Anything: Detailed Localized Image and Video Captioning
• Depth Anything 3: Recovering the Visual Space from Any Views
• Combining Large Language Models and Symbolic Reasoning for Multi-Robot Temporal Planning through Explainable Knowledge Bases
• NL2Plan: Robust LLM-Driven Planning from Minimal Text Descriptions
• CLIPort: What and Where Pathways for Robotic Manipulation
• VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models
• ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation
• Segment Anything
• SAM 2: Segment Anything in Images and Videos
• ConceptFusion: Open-set Multimodal 3D Mapping
• Neural Assembler: Learning to Generate Fine-Grained Robotic Assembly Instructions from Multi-View Images
• WorkBenchMark: A LEGO-Based Assembly Benchmark with an Assembly-by-Disassembly Baseline for the Smart Manufacturing League
• Fast Segment Anything

Related papers

• FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
• MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
• RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
• HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
• APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies
• Fine-tuning is Not Enough: A Parallel Framework for Collaborative Imitation and Reinforcement Learning in End-to-end Autonomous Driving