Towards Generalizable Visually Grounded Exploration of Household Devices
cs.AI
Submitted: 2026-09-01
Updated: 2026-09-01
Comments: Accepted to Findings of the Association for Computational Linguistics: EMNLP 2026
Code: https://github.com/BITHLP/VGEBench
License: http://creativecommons.org/licenses/by/4.0/
The gist: Recent advancements in Vision-Language Models (VLMs) have demonstrated impressive capabilities in static visual recognition and high-level semantic reasoning.
Terminology
Abstract
Recent advancements in Vision-Language Models (VLMs) have demonstrated impressive capabilities in static visual recognition and high-level semantic reasoning. However, current embodied exploration paradigms still heavily rely on imitation learning from human-annotated trajectories, which severely limits agents' generalization ability. The key bottleneck of realizing general autonomous embodied agents lies in Generalizable Visually Grounded Exploration: the ability to operate novel devices without manuals or specific training by actively grounding abstract world knowledge into fine-grained visual affordances. Yet, existing benchmarks fail to evaluate this capability: they generally rely on explicit documents and annotated trajectories, neglecting the dynamic Hypothesis-Interaction-Refinement process essential for functional device operation. To bridge this gap, we introduce VGEBench, a comprehensive benchmark designed to evaluate the generalizable visually grounded exploration capabilities of VLMs. Unlike static datasets, we construct a Logic-Driven State Machine framework. This framework simulates multi-turn interaction loops, compelling agents to achieve goals by active visual perception and feedback-driven correction. Experimental results demonstrate that existing VLMs face significant challenges in translating semantic knowledge into physical execution and maintaining long-horizon state tracking.
Sources
- On Evaluation of Embodied Navigation Agents
- Qwen3-VL Technical Report
- ShapeNet: An Information-Rich 3D Model Repository
- Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic
- PaLM-E: An Embodied Multimodal Language Model
- GAPartNet: Cross-Category Domain-Generalizable Object Perception and Manipulation via Generalizable and Actionable Parts
- Seed1.5-VL Technical Report
- MiMo-Embodied: X-Embodied Foundation Model Technical Report
- PhyBlock: A Progressive Benchmark for Physical Understanding and Planning via 3D Block Assembly
- BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation
- HomeBench: Evaluating LLMs in Smart Homes with Valid and Invalid Instructions Across Single and Multiple Devices
- Kosmos-2: Grounding Multimodal Large Language Models to the World
- Code as Policies: Language Model Programs for Embodied Control
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs
- AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents
- ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AI
- Voyager: An Open-Ended Embodied Agent with Large Language Models
- InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
- Ferret: Refer and Ground Anything Anywhere at Any Granularity
- PhysToolBench: Benchmarking Physical Tool Understanding for MLLMs
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
- VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection