FailBench: How Reliable are VLMs at Judging Robot Task Success?
cs.RO, cs.AI
Submitted: 2026-09-03
Updated: 2026-09-03
Project page: https://metric-ai-lab.github.io/failbench
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progress
- PhAIL: A Real-Robot VLA Benchmark and Distributional Methodology
- RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies
- Reliable and Scalable Robot Policy Evaluation with Imperfect Simulators
- Qwen3-VL Technical Report
- Automating Robot Failure Recovery Using Vision-Language Models With Optimized Prompts
- ManipulationNet: An Infrastructure for Benchmarking Real-World Robot Manipulation with Physical Skill Challenges and Embodied Multimodal Reasoning
- AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation
- Score the Steps, Not Just the Goal: VLM-Based Subgoal Evaluation for Robotic Manipulation
- RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shot
- Are We on the Right Way to Assessing LLM-as-a-Judge?
- Gemma 4 Technical Report
- I-FailSense: Towards General Robotic Failure Detection with Vision-Language Models
- SAFE: Multitask Failure Detection for Vision-Language-Action Models
- GUARD: Grounding Uncertainty and Ablation-Based Risk Detection for Diffusion-Based VLAs
- PRM-as-a-Judge: A Dense Evaluation Paradigm for Fine-Grained Robotic Auditing
- RoboReward: General-Purpose Vision-Language Reward Models for Robotics
- Learning Actionable Manipulation Recovery via Counterfactual Failure Synthesis
- LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
- Evaluating Real-World Robot Manipulation Policies in Simulation
Related papers
- FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
- MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
- RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
- HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
- APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies
- Fine-tuning is Not Enough: A Parallel Framework for Collaborative Imitation and Reinforcement Learning in End-to-end Autonomous Driving