Benchmark Designers Should "Train on the Test Set" to Expose Exploitable Non-Visual Shortcuts

The gist: Multimodal benchmark designers should proactively try to “game” their own benchmarks first as a key step in the development lifecycle—adopting rigorous diagnostic and debiasing procedures to systematically identify, quantify, and mitigate non-visual biases.

Introduction

Robust benchmarks are crucial for accurately evaluating Multimodal Large Language Models (MLLMs) but models can ace many multimodal benchmarks without strong visual understanding by exploiting biases, linguistic priors, and superficial patterns The uncomfortable truth is that models can ace multimodal benchmarks without strong visual understanding They exploit biases, linguistic priors, and superficial patterns A model can answer a visual question without looking at the image It can ground language without grounding perception, yet still achieve a high score This observation motivates the central argument of this paper: multimodal benchmark designers should proactively stress-test their creations for exploitable non-visual shortcuts We demonstrate that the most rigorous and useful stress test involves directly “training on the test set”—not to overfit, but to adversarially probe the test set for such intrinsic vulnerabilities

The Challenge: Non-Visual Shortcuts Undermine Multimodal Evaluation

Non-visual shortcuts occur when questions can be solved without utilizing the visual input This can happen when MLLMs exploit world knowledge acquired during linguistic pretraining or leverage statistical correlations within the question-answer pairs themselves The prevalence of such shortcuts leads to inflated performance metrics, misrepresents true visual understanding capabilities, and can misguide research by rewarding pattern matching over genuine multimodal reasoning

What Constitutes a Non-Visual Shortcut?

Shortcuts should be defined by their effect on the evaluation task, not their origin A pattern—whether reflecting natural world knowledge, real-world statistical regularities, or procedural generation artifacts—becomes an exploitable non-visual shortcut if and only if it renders the visual input redundant for a task designed to measure visual understanding This definition has important implications for how we interpret world knowledge in visual benchmarks

Non-visual Shortcuts from Knowledge

The first category of exploitable shortcuts arises from the extensive world knowledge embedded in LLMs during pretraining [30] As shown in Fig. 2, benchmarks like MMMU [34] and VideoMME [12] exhibit clear evidence of this vulnerability: models benefit more from scaling up the LLM backbone than from enabling visual inputs In contrast, VSI-Bench [31] shows negligible gains from LLM scaling in blind settings but substantial improvements when vision is enabled, demonstrating greater robustness to knowledge-based shortcuts

Non-visual Shortcuts from Statistical Correlations

Statistical biases manifest across diverse task types and benchmarks Counting tasks often exhibit severe long-tailed answer distribution skews; in VSI-Bench, over 50% of such questions have ground truth answer “3” or fewer, enabling a simple diagnostic model to achieve a high score by consistently predicting “2” Spatial relation tasks can show imbalanced answer frequencies, where certain object categories disproportionately appear as correct answers Many size estimation tasks naturally follow predictable log-normal distributions; the VSI-Bench room size task is heavily concentrated around typical room dimensions (logmu ≈ 17m2, logσ ≈ 2), enabling accurate predictions without seeing the room

Diagnosing Non-visual Shortcuts via Test-Set Stress-Testing

The foundational principle of TsT is to quantitatively estimate the extent to which benchmark questions can be answered using exclusively non-visual information present in the test set itself The TsT methodology yields two critical outputs: 1) Overall TsT Accuracy, which provides a global estimate of the benchmark’s “non-visual solvability” 2) Sample-Level Bias Score s(x), representing the diagnostic model’s confidence in the ground truth answer when x was in the validation fold

LLM-Based TsT Diagnostic

The TsT-LLM diagnostic applies the TsT framework using a powerful language model (e.g., Qwen2.5-7B [28]) as the diagnostic model For each fold in the k-fold procedure, we parameter-efficiently fine-tune the LLM using Low-Rank Adaptation (LoRA) [15] on the question and answer-choice text (ignoring all visual inputs) from the k−1 training folds This process repeats k times, yielding predictions for every sample from a model that was not exposed to that sample during training

RF-Based TsT Diagnostic

TsT-RF follows the same k-fold cross-validation procedure but uses a Random Forest classifier [6] trained on hand-crafted non-visual features For each sample x, we extract features fnv (x) designed to capture any information present at test time (excluding visual input) that might correlate with the correct answer TsT-RF offers two primary advantages: 1) Computational efficiency—training on CPU in minutes without GPU requirements enables rapid iteration during benchmark refinement 2) Interpretability—feature importance analysis (e.g., Gini importance [21]) reveals which specific non-visual cues drive exploitability

Empirical Validation: TsT Reveals Widespread Shortcut Susceptibility

Applying our TsT diagnostics to four prominent multimodal benchmarks reveals that non-visual shortcut susceptibility is both widespread and significant TsT-LLM Results show that for template-based benchmarks CV-Bench and VSI-Bench, TsT-LLM achieves dramatic gains of +33.3 and +31.4 points respectively, indicating that substantial fractions of these benchmarks can be “solved” by learning patterns in non-visual data alone TsT-RF Results show that on CV-Bench, TsT-RF achieves 75.5%—slightly exceeding TsT-LLM’s 73.

1. Improved diagnostic framework for identifying non-visual shortcuts: Implement a Test-set Stress-Test (TST) methodology that involves "fine-tuning a powerful Large Language Model (LLM) via k-fold cross-validation on exclusively the non-visual, textual inputs of the test set to unveil shortcut performance and derive a quantitative, sample-level bias score, s(x). This allows for the derivation of overall non-visual solvability and per-sample bias scores s(x) which are used as a foundation for targeted mitigation."

2. Improved mitigation strategy for benchmark refinement: Apply an Iterative Bias Pruning (I B P) procedure to systematically filter samples identified as highly biased according to s(x), ensuring that the process is adaptive by re-diagnosing the remaining set iteratively, because the removal of some highly biased samples can alter the statistical landscape of the remaining data.

3. More robust benchmark creation: Create VSI-Bench-Debiased by applying IBP guided by TST scores, resulting in a benchmark that has a significantly wider performance gap between vision-enabled and 'blind' MLLM configurations compared to the original, thereby creating a more reliable benchmark version.

4. Enhanced interpretability for designers: Utilize an RF-based diagnostic (TsT-RF) to provide interpretability—revealing which specific non-visual cues drive exploitability, such as showing that The obj val log mean feature dominates with importance 0.968, allowing designers to implement targeted mitigation strategies like removing questions about low-variance objects.