Failure Identification in Imitation Learning Via Statistical and Semantic Filtering

summary

The gist

The gist The introduction proposes FIDeL, a policy-independent failure detection module that uses vision-based anomaly detection combined with semantic filtering to distinguish between benign

In short

FIDeL is a policy-independent failure detection module for imitation learning policies that distinguishes between normal deviations and actual failures. It uses vision-based anomaly detection to find unusual states, aligns these states using optimal transport matching, and then employs a Vision-Language Model (VLM) for semantic filtering to accurately classify anomalies as genuine failures.

Key concepts

Optimal Transport Matching: This technique is used to align incoming observations with the compact representation of nominal demonstrations. It calculates the minimum cost required to transform a query feature into a key from the learned memory, providing a robust way to measure how different an observation is from what was expected during training.
Conformal Prediction (CP) Extension: This framework is extended to create dynamic thresholds for the anomaly detector. It helps set bounds that account for both temporal structure and variations in visual patches, ensuring that the system only flags states that are statistically significant deviations from normal behavior.
Vision-Language Model (VLM) Filtering: A VLM acts as a semantic filter to distinguish between benign anomalies and true failures. It is prompted with task context and visual evidence to classify an anomaly, helping the system avoid unnecessary interventions while reliably identifying genuine execution errors.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Federated Learning — explained in 17 episodes
Retrieval-Augmented Generation (RAG) — explained in 16 episodes
Interpretability — explained in 15 episodes
Quantum Fisher Information (QFI) — explained in 15 episodes
Sample Complexity — explained in 15 episodes

This episode discusses

The paper

Failure Identification in Imitation Learning Via Statistical and Semantic Filtering · Read on arXiv

Quentin Rolland, Fabrice Mayran de Chamisso, Jean-Baptiste Mouret

Universite Paris-Saclay · CEA List FactoryAI

Imitation learning (IL) policies in robotics deliver strong performance in controlled settings but remain brittle in real-world deployments: rare events such as hardware faults, defective parts, unexpected human actions, or any state that lies outside the training distribution can lead to failed executions. Vision-based Anomaly Detection (AD) methods emerged as an appropriate solution to detect these anomalous failure states but do not distinguish failures from benign deviations. We introduce FIDeL (Failure Identification in Demonstration Learning), a policy-independent failure detection module. Leveraging recent AD methods, FIDeL builds a compact representation of nominal demonstrations and aligns incoming observations via optimal transport matching to produce anomaly scores and heatmaps. Spatio-temporal thresholds are derived with an extension of conformal prediction, and a Vision-Language Model (VLM) performs semantic filtering to discriminate benign anomalies from genuine failures. We also introduce BotFails, a multimodal dataset of real-world tasks for failure detection in robotics. FIDeL consistently outperforms state-of-the-art baselines, yielding +5.30% percent AUROC in anomaly detection and +17.38% percent failure-detection accuracy on BotFails compared to existing methods.

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: Today's paper: "Failure Identification in Imitation Learning Via Statistical and Semantic Filtering".

Dev: The gist The introduction proposes FIDeL,

Rosa: First, who's behind it and why it matters.

Paper summary: Rosa: We're wrapping up on "Failure Identification in Imitation Learning Via Statistical and Semantic Filtering". The authors, Quentin Rolland, Fabrice Mayran de Chamisso, and Jean-Baptiste Mouret, are proposing FIDeL as a way to make robot monitoring more reliable.

Dev: Basically, this paper is about taking those imitation learning robots that work fine in the lab and figuring out how to actually spot when they fail out there in real life.

Taro: The main idea is using vision-based anomaly detection, but with a smart filter on top to make sure you're not flagging normal bumps as real failures.

Rosa: They call their system FIDeL, and it builds a memory of what normal robot actions look like from the training demonstrations.

Dev: It uses optimal transport matching to compare what’s happening now against that stored memory to get an anomaly score.

Taro: And then they use conformal prediction for thresholds, which means the system knows when a deviation is big enough to be worrying.

Rosa: But here's the kicker, they add a Vision-Language Model, or VLM, to semantically check if that weird state is actually a failure.

Dev: They found that this approach beats existing methods on their own dataset, BotFails, by showing better failure detection accuracy and anomaly detection scores.

Taro: The implication is that you don't just need a good policy; you also need a system robust enough to tell the difference between a little slip and something actually broken.

Rosa: It’s about building monitoring systems for robots that can actually interpret what they see when things go wrong in the messy real world.

Dev: But we have to keep an eye on the limitations, like how much variation in training data it needs and how much processing power that VLM filter uses.

Taro: Exactly, because if you don't have diverse demos, the system might just flag every tiny change as a disaster.

Rosa: So while FIDeL shows real promise for making robot monitoring more reliable, we still need to figure out how to make those filters run fast enough on the actual hardware.

Conclusion: Rosa: So we're wrapping up on "Failure Identification in Imitation Learning Via Statistical and Semantic Filtering". This paper is really about taking those imitation learning robots that work fine in the lab and figuring out how to actually spot when they fail out there in real life.

Dev: Exactly. They built a system, FIDeL, that uses vision-based anomaly detection combined with semantic filtering to separate real failures from normal bumps in the road when things go wrong during robot deployment.

Taro: The authors are Quentin Rolland and Fabrice Mayran de Chamisso and Jean-Baptiste Mouret. It’s interesting because they tackle that false positive problem where you get flagged for something that isn't actually a disaster.

Rosa: They do this by building a memory of what nominal robot actions look like from the training demonstrations, then using optimal transport matching to compare current observations against that memory to get anomaly scores.

Dev: And they use conformal prediction to set those thresholds, which means the system knows when a deviation is big enough to worry about, accounting for both how fast things are moving and where on the robot it's happening.

Taro: The key addition is this Vision-Language Model that does semantic filtering—it looks at the anomaly heatmaps and decides if what it sees is actually a failure or just something benign that happens all the time.

Rosa: It seems like this approach really balances robustness to natural task variations with sensitivity to those critical failure-related deviations we're looking for in these imitation learning systems.

Dev: But we have to keep an eye on the limitations, like how much variation in training data it needs and how much processing power that VLM filter uses for real-time operation.

Taro: That makes sense because if you only train it on one specific way to do a task, anything slightly different will look like an error.

Rosa: So while FIDeL shows real promise for making robot monitoring more reliable, we still need to figure out how to make those filters run fast enough on the actual hardware.

Dev: That makes sense. So this whole process of encoding demonstrations, aligning them with new observations using optimal transport, thresholding with conformal prediction, and finally filtering with a VLM is what they are proposing in "Failure Identification in Imitation Learning Via Statistical and Semantic Filtering".

Taro: It points toward a future where robot monitoring isn't just about how good the policy is on paper but also about recognizing and interpreting when things go wrong in practice.

More episodes

← Home