FragileFlow: Spectral Control of Correct-but-Fragile Predictions for Foundation Model Robustness

arXiv:2605.08896 · cs.CL, cs.AI, cs.LG · Submitted 2026-05-09 · Read on arXiv

cs.CL, cs.AI, cs.LG

Submitted: 2026-05-09

Updated: 2026-09-06

Comments: Added funding acknowledgments

License: http://creativecommons.org/licenses/by/4.0/

The gist: Robust adaptation of LLMs and VLMs is often evaluated by average accuracy or average consistency under perturbations.

Terminology

Abstract

Robust adaptation of LLMs and VLMs is often evaluated by average accuracy or average consistency under perturbations. However, these averages can hide a structured failure mode: a prediction may remain correct while probability mass already flows from particular true classes toward systematic wrong competitors near the decision boundary. In this paper, we formalize this phenomenon as margin-aware error flow and introduce FragileFlow, a plug-in regularizer that uses a calibrated margin buffer to identify correct-but-fragile predictions and organize their off-class probability mass into a class-wise vulnerable-risk matrix. Theoretically, we provide the first PAC-Bayes upper bound for this margin-aware error-flow object, showing how empirical spectral control yields a conservative route to deterministic worst-class robustness under a stability condition. Experiments on multiple-choice LLM benchmarks and few-shot CLIP adaptation show that FragileFlow consistently improves the proposed theory-facing risk measures over matched baselines, yields perturbed worst-class accuracy gains in most settings, and preserves clean accuracy across comparisons.

Sources

Related papers