JRDB-AVR: An Active Visual Reasoning Benchmark for Embodied Agents in Real-World Environments

arXiv:2609.35032 · cs.AI, cs.CV, cs.RO · Submitted 2026-09-28 · Read on arXiv

cs.AI, cs.CV, cs.RO

Submitted: 2026-09-28

Updated: 2026-09-28

Code: https://github.com/ControlNet/JRDB-AVR

Terminology

• Benchmark — 53× in this paper · explained in 2 episodes
• World Model — 36× in this paper · explained in 7 episodes
• Grounding — 23× in this paper · explained in 3 episodes
• Hallucination — 12× in this paper · explained in 3 episodes
• Accuracy — 7× in this paper · explained in 1 episode
• Ablation — 3× in this paper · explained in 1 episode
• Ablation study — 2× in this paper · explained in 1 episode
• Core — 2× in this paper · explained in 2 episodes
• Interpretability — 2× in this paper · explained in 15 episodes
• Temporal Reasoning — 2× in this paper · explained in 2 episodes
• Agentic Loop — 1× in this paper · explained in 1 episode
• Agentic System — 1× in this paper · explained in 2 episodes
• Chain-of-Thought Prompting — 1× in this paper · explained in 2 episodes
• Compositional Generalization — 1× in this paper · explained in 4 episodes
• Consistency — 1× in this paper · explained in 3 episodes
• Elias — 1× in this paper · explained in 2 episodes
• Foundation Model — 1× in this paper · explained in 3 episodes
• Foundation Models — 1× in this paper · explained in 7 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• Multimodal Reasoning — 1× in this paper · explained in 5 episodes
• Provenance — 1× in this paper · explained in 4 episodes
• Redundancy — 1× in this paper · explained in 2 episodes
• Reinforcement Learning — 1× in this paper · explained in 4 episodes
• Spatio-temporal grounding — 1× in this paper · explained in 2 episodes
• Structured Reasoning — 1× in this paper · explained in 3 episodes
• Trace — 1× in this paper · explained in 2 episodes
• Verifiability — 1× in this paper · explained in 2 episodes
• Verification — 1× in this paper · explained in 2 episodes
• Visual Grounding — 1× in this paper · explained in 2 episodes

Sources

• Qwen3-VL Technical Report
• Qwen2.5-VL Technical Report
• JRDB-Pose3D: A Multi-person 3D Human Pose and Shape Estimation Dataset for Robotics
• Memory-Anchored Multimodal Reasoning for Explainable Video Forensics
• An Empirical Study on How Video-LLMs Answer Video Questions
• ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving
• Mini-BEHAVIOR-Gran: Revealing U-Shaped Effects of Instruction Granularity on Language-Guided Embodied Agents
• What We Talk About When We Talk About LLM Planning: Evidence for Two Distinct Planning Abilities
• Explain Before You Answer: A Survey on Compositional Visual Reasoning
• VIEW2SPACE: Studying Multi-View Visual Reasoning from Sparse Observations
• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
• DrVideo: Document Retrieval Based Long Video Understanding
• JRDB-Traj: A Dataset and Benchmark for Trajectory Forecasting in Crowds
• HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
• ViperGPT: Visual Inference via Python Execution for Reasoning
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• MindCube: Spatial Mental Modeling from Limited Views
• VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos
• VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use
• Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
• VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection