Words Speak Louder Than Code: Investigating Cognitive Heuristics in LLM-Based Code Vulnerability Detection

summary

Video file (mp4)

The gist

The gist The first systematic investigation of cognitive heuristics in LLM-based code vulnerability detection reveals that all evaluated models are susceptible to these heuristics, and cognitive

In short

Researchers systematically investigated how cognitive heuristics—like the halo effect, framing effect, and anchoring effect—bias Large Language Models (LLMs) when detecting code vulnerabilities. The study found that all models are susceptible to these biases, which can lead to incorrect verdicts. Crucially, these biases do not improve detection capability but instead make models more or less likely to flag code as vulnerable.

Key concepts

Halo Effect
This is a cognitive shortcut where a positive impression of an entity in one area influences the evaluation of unrelated attributes. In this study, it means an LLM might trust code from a high-prestige author (like a principal engineer) more than code from a junior developer, regardless of the actual security flaws.
Framing Effect
This refers to how the way information is presented—the 'frame'—affects choices. For vulnerability detection, this means whether asking an LLM to identify 'violations' (negative framing) or verify 'compliance' (positive framing) changes its assessment of vulnerable code.
Anchoring Effect
This heuristic describes how an initial piece of information disproportionately influences subsequent judgments. In this context, it means that a prior analysis result, such as a static analyzer's verdict, can anchor the LLM's current assessment of new code.
Recall Gap (∆R) and Utility Index (UI)
These are custom metrics used to measure how much a cognitive heuristic shifts recall (finding true vulnerabilities) versus false positive rate. A positive UI means the heuristic is useful, but the study found that for most heuristics, this utility index is negative, meaning biases don't improve detection.

Terminology used across episodes

This episode discusses

The paper

Words Speak Louder Than Code: Investigating Cognitive Heuristics in LLM-Based Code Vulnerability Detection · Read on arXiv

BRAC University · Purdue University · University of Illinois Urbana-Champaign

Researchers and practitioners increasingly apply Large Language Models (LLMs) for automated vulnerability detection. Recent work has shown that LLMs are susceptible to the same cognitive heuristics that bias human judgment. Yet, no work has investigated whether these heuristics affect a model's assessment of code vulnerabilities. In this paper, we present the first systematic exploration of cognitive heuristics in LLM-driven code vulnerability detection. We introduce a controlled framework that holds the code fixed and only varies the surrounding context to trigger three cognitive heuristics: the halo effect through author attribution, the framing effect through task objectives and consequences, and the anchoring effect through prior analysis results. Within this framework, we evaluate eight LLMs across three programming languages. Our results demonstrate that all evaluated models are susceptible to these heuristics. Cross-model average susceptibility is highest for framing at 33.2%, followed by anchoring at 23.5% and halo at 18.4%. Vulnerabilities that require semantic reasoning for detection are more susceptible to cognitive heuristics than those identifiable through pattern matching. Furthermore, models often change their verdict from safe to vulnerable based on the cognitive condition, without accurately identifying the actual vulnerability. To highlight the practical impact, we demonstrate a proof-of-concept black-box cognitive attack that can suppress up to 97% of previously detected vulnerabilities. Together, these findings show that cognitive bias is a consistent and exploitable property of LLM-based vulnerability detection.

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: Today's paper: "Words Speak Louder Than Code".

Elias: The gist The first systematic investigation of cognitive heuristics in LLM-based code vulnerability detection reveals that all evaluated models are susceptible to these heuristics,

Nadia: First, who's behind it and why it matters.

Title and authors: Elias: First, the title itself, "Words Speak Louder Than Code," suggests that the context around the code—the words—can override what's actually in the code itself.

Nadia: It implies that sometimes a bad author tag or a scary task description can trick an LLM into making a wrong security call.

Elias: The authors are Asif Shahriar, Hongyu Cai, Hadjer Benkraouda, Gang Wang, and Z. Berkay Celik from BRAC University and Purdue University at the University of Illinois Urbana-Champaign.

Priya: It’s interesting that they’re focusing on how these context signals—like an author's reputation or a task description—can flip the model's security verdict.

Nadia: And what they want us to get from this is understanding if we can actually use these known cognitive biases to trick the AI into missing vulnerabilities or flagging harmless code incorrectly.

The paper's summary: Elias: The paper introduces a framework where they fix the actual source code and only change things around it, like commit messages or author names, to trigger these three specific heuristics.

Nadia: They evaluate eight different LLMs across three programming languages using this setup to see how susceptible they are to these biases.

Priya: What I’m hearing is that they found that all the models they tested were susceptible to these heuristics, and there was a consistent pattern of bias across them.

Elias: The numbers show the framing effect had the highest average susceptibility at thirty-three point two percent, followed by anchoring at twenty-three point five percent, and then the halo effect at eighteen point four percent <ref:2606.30587#pg1>.

Nadia: And they found that vulnerabilities that require deep reasoning, like tracking program state, were way more susceptible to these cognitive tricks than just things you can spot by looking for a simple pattern match.

The paper's improvements: Nadia: The authors point out that cognitive biases don't actually make the model better at finding bugs; instead, they just make it more or less willing to flag something as vulnerable, shifting both its recall and false positive rate by a similar amount.

Elias: That means the bias isn't improving accuracy in any direction; it’s just changing its willingness to say yes or no to a vulnerability verdict.

Priya: And they show that the direction of influence isn't always the same for every model; some models go against what you expect, especially with author attribution and prior analysis results.

Nadia: The paper demonstrates a practical exploit by showing a black-box attack that can suppress up to ninety-seven percent of previously detected vulnerabilities in a real CI/CD environment just by faking commit metadata and scan reports.

Conclusion: Elias: So, the main conclusion is that cognitive heuristics are a consistent and exploitable property of LLM-based vulnerability detection; an attacker can strategically place context to deceive the model.

Nadia: It shows that prompt-based defenses aren't sufficient against these attacks because they don't eliminate the suppression effect on most models.

Priya: The paper also makes it clear that we need to focus our testing on those deep logic checks, like tracking program state, because those classes are consistently more susceptible to these heuristics than surface-level ones.

Elias: Ultimately, the implication is that security analysis needs to account for this possibility that context can systematically bias the model's objective verdict.

Nadia: What we’ve heard today about "Words Speak Louder Than Code" is that even advanced AI security tools have these human cognitive weaknesses, and those weaknesses are measurable and exploitable in a real workflow.

More episodes

← Home