Dataset Biases and Shortcut Learning in Motion-Based AI-Generated Video Detection

arXiv:2607.00948 · cs.CV · Submitted 2026-07-01 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Dataset Biases and Shortcut Learning in Motion-Based AI-Generated Video Detection".

Jane: The visual quality of AI-generated videos has improved drastically, making it increasingly difficult for humans to distinguish between real and synthetic media, necessitating robust detection methods.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So, let's talk about who wrote this paper. The title itself, "Dataset Biases and Shortcut Learning in Motion-Based AI-Generated Video Detection," really tells you exactly what they’re focusing on—the data biases and those shortcuts models take instead of learning the actual motion differences.

Jane: And the authors are Joren Michels, Lode Jorissen, and Nick Michiels from the Digital Future Lab at Hasselt University. It’s interesting to see research coming from a university setting that is so focused on these kinds of deep methodological critiques in AI.

Lu: Their work connects this specific issue to earlier findings about image detection where models exploit dataset biases, like how one dataset for tampered images had a bias related to JPEG compression quality factors (<ref:2607.00948#pg2>). This paper applies that same logic specifically to motion detection using different types of AI video generation.

Meng: I’m thinking about the practical implications here, Lu. If we can't trust the reported accuracy because of these biases, then deploying any motion detector in a production system becomes risky because we don't know if it’s actually working on real videos or just on a biased subset.

Lalam: It means that for AI video authentication tools to be truly useful, they need to be built with awareness of the data sources they are trained and tested on, rather than just trusting the highest reported scores without scrutiny.

The paper's summary: Tom: Moving into what the paper actually says, it summarizes that four state-of-the-art motion detectors—D3, ReStraV, Over-Coherence, and NSG-VD—all show nearperfect performance on their specific evaluation datasets. However, the core finding is that a substantial portion of that high performance is actually due to exploiting preprocessing and sampling biases inherent in those evaluations.

Jane: So they found that these detectors are highly sensitive to motion patterns unique to those datasets, where AI-generated videos tend to have less inter-frame movement compared to real videos, which gives the detectors an easy way out.

Lu: They systematically analyzed specific sampling and preprocessing issues for each detector; for instance, D3 had a bias where real videos were saved at three frames per second while others were sampled at much higher rates, leading to duplicated frames that D3 could exploit (<ref:2607.00948#pg1>).

Meng: That detail about frame duplication is really concrete. It shows that the reported AUC of ninety-seven point seven two for D3 isn't just a measure of motion detection skill; it’s partly a measure of how well it handles that specific, flawed data pipeline <ref:2607.00948#pg0>.

Lalam: This highlights how subtle technical details in the evaluation process can completely skew our perception of an AI system's capability, and we need to pay attention to those details when assessing new tech.

The paper's improvements: Tom: Now for the part where they suggest fixes, because it’s not just about pointing out flaws; they propose concrete ways to address them. They suggest using dataset rebalancing and applying simple spatial augmentations to observe severe performance degradation across all models when these biases are introduced.

Jane: The proposed improvements focus on making the detectors more resilient by forcing them to work with data that better reflects real-world motion characteristics, rather than relying on the artificially smooth movement seen in synthetic videos.

Lu: The paper suggests incorporating a hybrid feature strategy, meaning instead of just looking at simple second-order differences of frame embeddings, we should combine that with frequency domain analysis to focus on mid- to high-frequency artifacts, similar to what WaveRep does (<ref:2607.00948#pg1>).

Meng: From a practical standpoint, I think incorporating frequency analysis is smart because it might capture the actual noise introduced by the generative process, which is less dependent on how we sample the video frames during testing.

Lalam: If we can make these systems more robust against sampling biases and resolution artifacts, it opens up possibilities for building detection tools that are reliable regardless of which specific generative model was used to create the video.

Conclusion: Tom: So, to wrap up this discussion on "Dataset Biases and Shortcut Learning in Motion-Based AI-Generated Video Detection," the paper’s main implication is that reported high performance in motion detectors often masks deep dependencies on flawed evaluation protocols rather than true motion recognition abilities.

Jane: Essentially, they are showing that we need to rigorously test these detectors on data sets that truly represent the real world, or we risk deploying systems that only work because of artificial data shortcuts.

Lu: The findings suggest a path forward by moving away from dataset-specific performance metrics and toward more invariant feature learning strategies, like those involving frequency domain analysis mentioned in the paper.

Meng: I see this as a signal that our engineering focus needs to shift from just chasing the highest AUC scores on specific benchmarks to designing systems that are resilient against these kinds of data artifacts during training and testing.

Lalam: This work really underscores the need for transparency in how we evaluate AI video detection tools so that they become trustworthy instruments for media forensics moving forward.

Digital Future Lab · Flanders Make · Hasselt University

cs.CV

Submitted: 2026-07-01

Updated: 2026-10-06

Code: https://github.com/FujitsuResearch/training-free-detection-oftext-to-video-generations-via-over-coherence

Importance score: 76/100

The gist: The visual quality of AI-generated videos has improved drastically, making it increasingly difficult for humans to distinguish between real and synthetic media, necessitating robust detection methods.

Key concepts

Preprocessing Biases
These are systematic errors introduced during the preparation of evaluation data, like changing frame rates or downscaling video resolution before testing. Detectors can learn to exploit these specific preparation quirks instead of detecting actual AI-generated video flaws.
Sampling Biases
This occurs when the way data points are selected for testing is not representative of the real world. For example, in one detector, real videos were sampled at a lower frame rate than original data, creating artificial motion patterns that the detector can easily recognize.
Resolution-Dependent Artifacts
This bias relates to how video quality is handled during detection. When high-resolution videos are resized and compressed (like JPEG), certain visual details disappear. Detectors can become reliant on these resolution artifacts instead of genuine motion differences between real and fake videos.

Terminology

Summary

The visual quality of AI-generated videos has improved drastically, making it increasingly difficult for humans to distinguish between real and synthetic media, necessitating robust detection methods. This work evaluates four state-of-the-art motion-based AI-generated video detectors, revealing that their reported performance is significantly inflated by preprocessing and sampling biases inherent in their evaluation datasets.

The gist

Significant preprocessing and sampling biases in motion-based AI-generated video detectors account for a substantial portion of their reported performance, with detectors exhibiting high sensitivity to motion patterns specific to their evaluation datasets where AI-generated videos generally exhibit less inter-frame movement than real videos.

Evaluation of Motion Detectors and Bias Exploitation

The study focused on four recent motion-based detectors: D3 (ICCV 2025), ReStraV (NeurIPS 2025), Over-Coherence (WACV 2026), and NSG-VD (NeurIPS 2025). Reported results for these approaches indicated nearperfect performance on their evaluation datasets (e.g., D3: 97.72 AUC, ReStraV: 98.81 mAP, NSG-VD: 96.14 AUC). However, the research systematically analyzed the extent of biases and found that a substantial portion of the reported performance of these detectors stems from exploiting such biases, rather than capturing motion-related differences between AI-generated and real videos.

Sampling Biases in Specific Detectors

The analysis identified specific sampling biases within three of the four detectors:

  1. For D3, it was observed that real videos in the GenVideo dataset were saved at 3 frames per second (fps), while original MSR-VTT data was between 25 and 30 fps. This introduced a major sampling bias where real videos at 3 fps are upsampled to 8 fps, resulting in frames being duplicated (up to three consecutive duplicate frames per video). This artificially introduces peaks in second-order differences that D3 can exploit.

  2. For ReStraV, videos with lower fps than 12 were duplicated to reach this rate, and videos shorter than 2 seconds had their final frame repeated until the required number of frames was obtained. The most noticeable change was that the minimum distance value will be 0 if duplicate frames are present, a pattern the MLP can easily exploit.

  3. For NSG-VD, sampling involved extracting eight frames uniformly spaced across the entire video sequence, establishing a correlation between total video length and temporal distance between consecutive sampled frames. Furthermore, it was noted that the division by ∆t in Equation 1 is not performed, mathematically treating the temporal gap as identical for all videos.

Resolution-Dependent Artifacts in NSG-VD

The NSG-VD detector exhibited a critical bias related to video resolution and preprocessing pipelines. The pipeline involved extracting frames at native resolution and resizing them to 256x256 pixels, followed by JPEG encoding. This introduced a resolution-dependent distortion: when high-resolution videos are significantly downscaled, the JPEG and H.264 quantization blocks are suppressed. The study demonstrated that when using the original pipeline (JPEG compression before downscaling), spectral peaks showed clear differences between real and generated videos; however, applying JPEG compression after downscaling resulted in the spectral peaks become indistinguishable between real and generated videos, indicating reliance on resolution-related artifacts rather than true discriminative features.

Impact of Motion Biases on Generalizability

The researchers analyzed five recent datasets (GenVideo [12], GenBuster++ [21], WaveRep [14], Over-Coherence [24], and VidProM [27]) to examine motion biases. A simple metric—the standard deviation of mean pixel values over all frames of a video—revealed a pattern: In the GenVideo, VidProM, Over-Coherence and Waverep datasets, real videos tend to have more erratic motion patterns, while AI-generated videos have smoother motion patterns. This bias was only balanced in the GenBuster++ dataset. When detectors were re-evaluated on temporally (re-)balanced datasets, their performance degraded significantly. Crucially, when the evaluation was performed on a dataset that did not contain this motion bias (GenBuster++), all three detectors drop to random-level performance, suggesting their high accuracy is dependent on these dataset biases rather than inherent properties of AI-generated videos.

Robustness and Frequency-Based Alternatives

To test if the observed problems were specific to motion-based detectors, the researchers evaluated a non-motion-based detector, WaveRep [14], which uses an augmentation strategy that "replaces specific low-frequency bands in AI-generated videos with real counterparts to force the model to focus on mid- to high-frequency artifacts.

Improvements for AI systems

As a fastidious and diligent researcher, I have analyzed this paper to identify critical vulnerabilities in current motion-based AI-generated video detectors. The core improvement strategy involves moving away from reliance on dataset-specific motion biases toward more robust, generalizable feature extraction methods.

Here are the specific improvements for AI systems and what those improved systems can achieve:

The primary goal is to develop a detector that is invariant to dataset sampling biases and resolution artifacts, leading to higher generalization across diverse generative models (e.g., Sora, GenVideo).

  1. Incorporate a Hybrid Feature Strategy:

  2. Retrain or fine-tune motion-based detectors (like D3 and ReStraV) using a multi-faceted feature set instead of relying solely on second-order differences of frame embeddings. This hybrid approach should combine:

  3. Frequency Domain Analysis (Mid/High Frequencies): Explicitly incorporate WaveRep's strategy—training models to focus on mid- to high-frequency artifacts—to capture generative noise that is less susceptible to simple motion sampling biases.

  4. Resolution and Encoding Invariance: Implement a preprocessing pipeline that normalizes or explicitly accounts for JPEG/H.264 quantization blocks (as shown in Section 3.3), ensuring the detector does not rely on artifacts introduced during downscaling or compression, which currently cause severe performance drops in NSG-VD.

  5. Motion Bias Mitigation via Data Rebalancing: For any motion-based detector, mandate a rebalancing strategy (e.g., binning based on standard deviation of mean pixel values) to ensure the training and evaluation sets have a statistically similar distribution of motion characteristics between real and synthetic videos (as demonstrated in Section 4.2).

  6. Establish Robustness via Augmentation Testing: Before deployment, subject any detector to targeted spatial augmentations (like a 7x7 Gaussian blur) that are known to cause significant performance drops in current models, ensuring the detector's decision boundary is not overly reliant on trivial inter-frame encoding differences.

The resulting improved AI system (a generalized motion-based video detector) will be capable of:

  1. Detecting AI-generated videos with significantly higher accuracy and robustness across a wider variety of generative models, moving beyond the limitations imposed by specific training datasets (e.g., GenVideo or VidProM).

  2. Maintaining strong performance even when evaluated on unbiased datasets that lack the specific motion patterns exploited by current state-of-the-art detectors, as demonstrated by the frequency-based detector (WaveRep) maintaining high accuracy across all tested sets.

  3. Being less susceptible to adversarial attacks involving simple preprocessing steps like JPEG compression or resolution changes, making it more reliable for real-world deployment in media forensics.

  4. Providing a more generalizable detection signal that reflects inherent differences between real and synthetic video generation physics, rather than superficial artifacts in the evaluation protocol.

Sources

Related papers