From Lab to Reality: A Practical Evaluation of Deep Learning Models and LLMs for Vulnerability Detection

arXiv:2512.10485 · cs.CR, cs.LG, cs.SE · Submitted 2026-08-17 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio.

Tom: Next we'll be talking about the paper "From Lab to Reality: A Practical Evaluation of Deep Learning Models and LLMs for Vulnerability Detection".

Jane: The paper was written by Chaomeng Lu and Bert Lagaisse from KU Leuven.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Summary: Tom: The core finding is that current representation methods, both graph-based and token-based, really struggle to separate vulnerable code from non-vulnerable code.

Jane: When they look at the t-SNE visualizations in Figure 3a and 3b, the separation isn't clean; there’s heavy overlap, especially on datasets like Juliet or ICVul.

Meng: That means that even if an AI model is trained to spot a pattern, it’s not robust enough to confidently say whether a specific function is vulnerable or not.

Lu: It suggests the models are memorizing specific patterns from the data rather than understanding the actual security semantics of the code itself.

Tom: And this isn't just a single failure; if you train them on one dataset, they barely work on another. For instance, LineVul trained on Juliet achieves ninety-five point nine eight percent accuracy but then drops to only thirty-eight point seven seven F1-score when tested on ICVul.

Jane: That’s a massive drop in performance for a model that looks so confident during training, illustrating the lack of cross-dataset generalization we’ve been talking about.

Meng: In deployment, where you need consistent reliability, that kind of performance variance is unacceptable; it means the model just fails to transfer its knowledge effectively across different codebases.

Lalam: It shows that our current AI systems are fantastic at solving textbook problems but terrible at handling the ambiguity and diversity of everyday programming challenges.

Tom: So, we've established that models fail to generalize and fail to represent vulnerabilities cleanly; let's move on to how they built a test environment that actually highlights these failures.

Improvements: Jane: The paper introduces this new, small-scale dataset called VentiVul, which is a completely out-of-distribution set.

Tom: It’s not synthetic; the authors manually collected twenty recently fixed vulnerabilities from the Linux kernel that were reported in May of two thousand twenty-five.

Meng: That manual curation is key, because it ensures we aren're testing against real, current security issues rather than historical or artificial ones.

Lu: I find the shift to VentiVul fascinating because it forces us to move away from the comforting predictability of BigVul and Juliet toward an evaluation that reflects actual current software engineering practices.

Lalam: It’s a powerful example of moving away from idealized training data toward real-world operational context, which is a massive cultural win for how we approach AI testing.

Tom: And this isn' testbed is used to test two specific modes: "Whole-File" and "Function-Pair."

Jane: The Function-Pair setup shows the model a function before it was fixed, and then its corresponding patched version, allowing us to measure if the model can tell which one is vulnerable.

Meng: That’s a very precise test; it asks the AI not just "is this code bad?" but "did this specific semantic change fix the problem?"

Lu: Imagine if we could build an automated system using that Function-Pair logic to identify exactly where a fix failed to address a core vulnerability—the potential for auditing is enormous.

Tom: The authors found that while LLMs do better at this patch-level reasoning, the whole file evaluation showed extreme biases.

Jane: We saw some models going all negative or all positive, meaning they either missed everything or flagged everything as dangerous.

Meng: For an enterprise deployment, you want a model that is consistently accurate in the middle range, not one that completely ignores half of its workload.

Lalam: The shift from "Whole-File" to "Function-Pair" tells us that true understanding requires localized semantic comparison, not just general structural analysis.

Tom: That leads perfectly into how they think we can fix this problem.

Improvements: Jane: The authors suggest several ways forward, fundamentally tied to the data and the representations themselves.

Tom: They point out that dataset quality is far more important than just size, so we need to focus on getting high-quality, accurately labeled data.

Meng: From an engineering standpoint, this means industry needs to stop relying on massive but noisy datasets and start focusing on small but perfectly clean ones for transfer learning.

Lu: And the researchers are also pushing for better code representations—not just generic token sequences or graph structures, but something that can capture those deeper, contextual data flows.

Jane: They noted that models like ReVeal underperform on noisy datasets, showing that architecture alone doesn' not guarantee generalization.

Meng: It seems the industry needs a new standard for "good" representation, something robust against label noise and structural inconsistencies across different projects.

Tom: This all ties back to the limitations of current AI; we are optimizing for lab performance instead of real-world applicability.

Lu: The idea is that our next generation of code understanding must go beyond just matching patterns to truly modeling the semantic intent behind a fix or a vulnerability.

Jane: It's about acknowledging that the problem is not just poor data, but also flawed assumptions about what makes code "vulnerable."

Meng: If we have better representations, do you think we could achieve consistent performance even on those real-world datasets like ICVul?

Lalam: The future of AI needs to be less of a pattern matcher and more of a contextual reasoner that respects the complexity and diversity inherent in the code it analyzes.

Tom: It’s clear that we need better tools, not just better training runs.

Conclusion: Jane: So, to wrap up on "From Lab to Reality: A Practical Evaluation of Deep Learning Models and LLMs for Vulnerability Detection," the main message is that current AI models are not ready for real-world deployment.

Tom: They are great at passing standardized tests but struggle severely when they fail to find new or subtle vulnerabilities in a complex, messy codebase.

Meng: The practical implication here is that if we want to use this technology for automated security auditing, we must design the entire pipeline around high-quality data and robust representations.

Lu: I’m excited about the potential for building specialized, CWE-specific models based on this research that could see how far we can push beyond these current limitations.

Jane: The authors are calling for a move away from simply being satisfied with benchmark results and toward a rigorous, deployment-oriented framework.

Tom: It’s a powerful reminder that the difference between lab success and real-world applicability is significant, especially when it comes to security.

Lalam: It's about building an AI that can handle uncertainty, not just one that confidently picks the right answer in a world of possible failures.

Tom: We have some big ideas here for how to improve our code analysis tools; we'll be back after the break to talk about where the industry is headed next.

Chaomeng Lu, Bert Lagaisse

KU Leuven

cs.CR, cs.LG, cs.SE

Submitted: 2026-08-17

Updated: 2026-08-18

Code: https://github.com/Chaomeng-Lu/A-Practical-Evaluation-of-Deep-Learning-

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 80/100

Key concepts

Lack of Cross-Dataset Generalization: Models trained on one dataset, like Juliet, perform well but fail significantly when tested on another dataset, such as ICVul. This shows the models are memorizing specific patterns rather than understanding the actual security semantics of code.
VentiVul Dataset: A new, small-scale test dataset manually curated by the authors. It contains twenty recently fixed vulnerabilities from the Linux kernel, providing a real-world evaluation that moves away from historical or artificial training data.
Function-Pair Evaluation: A testing method where the model sees a function before it was fixed and then its corresponding patched version. This allows researchers to measure if the AI can specifically identify which semantic change solved the vulnerability.
Code Representation Methods: The way AI models view code, whether using token sequences or graph structures. The discussion highlights that current methods struggle to cleanly separate vulnerable code from non-vulnerable code, requiring a better representation that captures deeper contextual data flows.

Terminology

Benchmark — 47× in this paper · explained in 2 episodes
Accuracy — 31× in this paper · explained in 1 episode
Embeddings — 15× in this paper · explained in 2 episodes
Generalization — 15× in this paper · explained in 13 episodes
Robustness — 10× in this paper · explained in 5 episodes
Transformer — 7× in this paper · explained in 2 episodes
International Conference on Software Engineering — 6× in this paper
International Conference on Mining Software Repositories — 5× in this paper
Transferability — 5× in this paper · explained in 4 episodes
Generalizability — 4× in this paper · explained in 6 episodes
Core — 3× in this paper · explained in 2 episodes
Large Language Models (LLMs) — 3× in this paper · explained in 6 episodes
Consistency — 2× in this paper · explained in 3 episodes
GNNs — 2× in this paper · explained in 2 episodes
Graph Neural Network (GNN) — 2× in this paper · explained in 3 episodes
Representation Learning — 2× in this paper · explained in 2 episodes
Benchmarking — 1× in this paper · explained in 4 episodes
Feature Engineering — 1× in this paper · explained in 3 episodes
Feature Extraction — 1× in this paper · explained in 2 episodes
Fine-Tuning — 1× in this paper · explained in 5 episodes
In-Context Learning — 1× in this paper · explained in 4 episodes
Interpretability — 1× in this paper · explained in 15 episodes
Semantic Similarity — 1× in this paper · explained in 2 episodes
Trace — 1× in this paper · explained in 2 episodes
Transformer Models — 1× in this paper · explained in 2 episodes

Summary

Summary

This paper presents a systematic, deployment-oriented evaluation of deep learning (DL) models and large language models (LLMs) for software vulnerability detection, aiming to bridge the gap between academic benchmark performance and real-world applicability. The authors state their primary motivation: "DL–based vulnerability detection models have seldom been rigorously evaluated in deployment-oriented, realistic settings involving novel or previously unseen vulnerabilities. Public datasets frequently rely on synthetic or heuristically labeled samples, which may fail to capture the full diversity and complexity of real-world vulnerabilities [11, 21]. Moreover, most existing studies adopt a closed-world assumption, where training and test data are drawn from similar distributions, often leading to overly optimistic performance estimates [6, 7]."

The study addresses three research questions: RQ1 asks How effectively do current code representation methods capture distinctions between vulnerable and non-vulnerable code?; RQ2 asks What factors influence the performance of current deep learning models on benchmark vulnerability datasets in controlled lab settings?; and RQ3 asks How well do current deep learning and large language models generalize to detecting novel vulnerabilities in deployment settings?

The authors select two representative DL models—ReVeal (graph-based, using Joern-derived structural information with Word2Vec embeddings) and LineVul (token-based, using CodeBERT Tokenizer with a transformer-based classifier)—and evaluate them on four benchmark datasets: Juliet (synthetic, 253,002 functions, 37% vulnerable ratio, 100% label accuracy), Devign (manual, 27,318 functions, 46% vulnerable ratio, 80% label accuracy), BigVul (patches, 188,636 functions, 6% vulnerable ratio, 25-54% label accuracy), and ICVul (patches, 15,396 functions, 41% vulnerable ratio, 92% label accuracy). They also construct a new out-of-distribution dataset, VentiVul, comprising 20 recently (May 2025) fixed vulnerabilities from the Linux kernel, with 25 function pairs (vulnerable and patched versions) spanning 21 distinct.c source files and 835 unrelated functions.

For RQ1, the authors use t-SNE visualization and centroid distance analysis on embeddings from both representation methods. They find that GNN- and CodeBERT-based embeddings show limited and inconsistent separation between vulnerable and non-vulnerable functions across datasets. Specifically, GNN-based embeddings provide limited class separability with heavy overlaps in Juliet and ICVul (centroid distances 0.0242 and 0.0057), while BigVul shows slightly larger separation (0.1612) but lacks distinct clusters. CodeBERT embeddings yield somewhat better separation on Devign and BigVul but inconsistently. The authors also find that Centroid distances across vulnerability types are erratic and inconsistent, indicating unreliable feature-level distinctions from both embedding methods, and that Realistic datasets like ICVul have smaller centroid distances than synthetic ones like Juliet, due to higher code similarity and differing CWE type distributions.

For RQ2, the authors conduct cross-dataset experiments where each model is trained on one dataset and tested on all others. Results show that Model performance is highly dataset-dependent. Within-dataset testing yields strong results, but cross-dataset performance drops sharply, especially for models trained on Juliet or BigVul. For example, LineVul trained on Juliet achieves 95.98% accuracy and 94.52 F1 on Juliet but drops to 38.77 F1 on ICVul; ReVeal trained on BigVul drops to only 0.5 F1 on Devign. The authors identify key factors: "Label quality, dataset size, type, and balance critically influence cross-dataset generalization. High-quality, balanced datasets enable transferable learning, while synthetic datasets show limited real-world transferability, and noisy or imbalanced datasets reduce robustness and bias models toward majority classes. They also note that Model architecture alone does not ensure generalization. Both LineVul and ReVeal degrade similarly across datasets, indicating current representations fail to capture universally transferable vulnerability patterns."

For RQ3, the authors evaluate the trained DL models (in inference-only mode) and four LLMs (Claude 3.5 Sonnet, GPT-o3-mini, GPT-4o, GPT-5) on VentiVul using two novel evaluation settings: Whole-File (input is all functions from vulnerable files, each labeled) and Function-Pair (input is a pair of vulnerable and patched versions of a function). In the Whole-File setting, DL models exhibit extreme biases: ReVeal (BigVul) achieves 89.06% accuracy but with 4% recall and 2.17 F1 due to all-negative predictions; ReVeal (Devign) and LineVul (Devign) achieve 100% recall but poor precision (3%), yielding F1-scores of 6.27 and 5.73. LLMs achieve high accuracy (above 92%) but extremely low F1-scores (0–4.9) because they predict nearly all samples as non-vulnerable. The authors conclude: "DL models exhibit extreme biases in Whole-File evaluation, producing either all-negative or all-positive predictions. LLMs achieve high accuracy but extremely low F1 due to predicting mostly non-vulnerable samples, limiting practical usefulness."

In the Function-Pair setting, most DL models fail to differentiate before-fix and after-fix functions. ReVeal (Devign) and LineVul (Devign) detect all 25 vulnerable functions but differentiate zero pairs. LineVul (ICVul) is the only DL model that differentiates three pairs correctly. LLMs show better patch-level reasoning: Claude 3.5 Sonnet detects 13 vulnerabilities and differentiates 1 pair; GPT-o3-mini identifies only 4 vulnerabilities but differentiates 3 pairs; GPT-4o and GPT-5 each detect 13 vulnerabilities and differentiate 5 and 6 pairs, respectively. The authors state: "DL models largely fail to differentiate before-fix and after-fix functions, showing minimal sensitivity to semantic changes, with LineVul (ICVul) being a partial exception. LLMs outperform traditional DL models in patch-level reasoning."

The paper's key contributions are: analyzing code representations from ReVeal and LineVul across four datasets revealing limitations in separating vulnerable and non-vulnerable code; investigating cross-dataset generalization and uncovering factors impacting robustness and metric reliability; evaluating DL models and LLMs on VentiVul showing significant performance degradation; and manually constructing a high-quality out-of-distribution dataset with a deployment-oriented evaluation framework introducing Whole-File and Function-Pair modes.

The discussion highlights that current vulnerability detectors optimize for dataset-specific correlations rather than generalizable semantics, and that their inability to recognize code changes reflects a fundamental mismatch between benchmark training and deployment conditions. The authors recommend future work focus on advancing code representations and creating balanced, high-quality datasets to bridge the gap between lab and reality settings.

Improvements for AI systems

Based on the paper's findings, here are the specific improvements I can make to AI systems for vulnerability detection:

Improvement: Implement a time-aware confidence scoring mechanism that flags predictions on code samples from time periods or codebases significantly different from training data.
What the improved system can do: When deployed on new code (e.g., Linux kernel code from May 2025), the system will automatically detect distribution shift and either abstain from prediction or clearly indicate low confidence, rather than producing misleading high-confidence outputs.
Improvement: Add a contrastive learning objective that explicitly trains the model to differentiate between before-fix (vulnerable) and after-fix (patched) versions of the same function.
What the improved system can do: Instead of just classifying a function as vulnerable/non-vulnerable, the system can now reason about why a code change fixes a vulnerability. This addresses the paper's finding that current models assign nearly identical scores to before-fix and after-fix functions (Figure 5).
Improvement: Restructure the output layer to jointly predict (a) binary vulnerability status and (b) vulnerability type (CWE category), with a shared representation layer.
What the improved system can do: The system can distinguish between different vulnerability types (e.g., stack corruption vs. use-after-free) rather than treating all vulnerabilities as one class. This addresses the paper's finding that centroid distances between different CWE types often exceed those between vulnerable and non-vulnerable code.
Improvement: Add a noise-robust loss function (e.g., symmetric cross-entropy or noise transition matrix estimation) during training, specifically for datasets like BigVul with known label accuracy of only 25-54%.
What the improved system can do: The system will be less likely to memorize spurious patterns from mislabeled samples, improving cross-dataset generalization. This directly addresses the paper's finding that models trained on noisy datasets (BigVul) show severe performance degradation on other datasets.
Improvement: Implement post-training threshold optimization using precision-recall curves rather than default 0.5 threshold, with automatic adjustment based on expected deployment class distribution.
What the improved system can do: The system will avoid the all-negative or all-positive prediction biases observed in the Whole-File evaluation (Table 6), where models achieved high accuracy but near-zero F1 scores due to extreme class imbalance.
Improvement: Add a verification layer that cross-checks function-level predictions against file-level context, flagging inconsistencies (e.g., a function predicted vulnerable in isolation but non-vulnerable when evaluated with its surrounding file).
What the improved system can do: The system will provide more reliable predictions by considering both local function semantics and broader file context, addressing the paper's finding that models perform differently under Whole-File vs. Function-Pair evaluation settings.
Improvement: During inference, compute a semantic change score that measures how much the model's internal representation changes when comparing before-fix and after-fix versions of the same function.
What the improved system can do: The system can explicitly report whether a code change meaningfully alters its vulnerability assessment, enabling better human-AI collaboration in code review workflows. This addresses the finding that most models show minimal sensitivity to code fixes.
Improvement: Use a meta-learning approach that weights training samples based on estimated label quality, with higher weights for samples from high-quality datasets (e.g., ICVul with 92% label accuracy) and lower weights for noisy datasets (e.g., BigVul).
What the improved system can do: The system will learn more transferable vulnerability patterns rather than overfitting to dataset-specific artifacts, improving performance on novel, out-of-distribution vulnerabilities like those in VentiVul.