LLMs for Qualitative Data Analysis Fail on Security-specificComments in Human Experiments

summary

Video file (mp4)

The gist

The gist: LLMs for Qualitative Data Analysis Fail on Security-specific Comments in Human Experiments shows that large language models cannot reliably replace human annotators for security-relevant

In short

The study tested if Large Language Models (LLMs) could automatically label technical security comments made by humans during code analysis experiments. Despite using detailed codebooks and conflicting examples, LLMs failed to reliably match human annotators for security-specific coding tasks. The conclusion is that LLMs cannot yet replace human experts in this domain.

Key concepts

Thematic Analysis of Free-Text Justifications

This involves deeply understanding the reasons or justifications written by people in open text, such as comments on code. The goal is to find recurring themes or patterns within these justifications to gain qualitative insights into human behavior and decision-making during experiments.

Security-Specific Annotations

These are specialized coding tasks where the LLM must identify specific technical elements within comments, such as mentioning particular security codes, lines of code, or specific security keywords. This requires a deep understanding of the context beyond general sentiment analysis.

Prompt Engineering Strategy

This is the process of carefully designing instructions given to an LLM to guide its output toward a desired result. The researchers tried different methods, including using emerging codes, formal definitions from codebooks, and presenting conflicting examples to see how these inputs affected the LLM's ability to annotate correctly.

Terminology used across episodes

• Reinforcement Learning (RL) — explained in 25 episodes
• Sample Complexity — explained in 19 episodes
• Federated Learning — explained in 17 episodes
• Retrieval-Augmented Generation (RAG) — explained in 17 episodes
• Quantum Fisher Information (QFI) — explained in 16 episodes
• Interpretability — explained in 15 episodes

This episode discusses

• LLMs for Qualitative Data Analysis Fail on Security-specificComments in Human Experiments · Paper Radio
• LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis
• DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
• Scalable Qualitative Coding with LLMs: Chain-of-Thought Reasoning Matches Human Performance in Some Hermeneutic Tasks
• GPT-4o System Card
• VulDeePecker: A Deep Learning-Based System for Vulnerability Detection
• It's Difficult to be Neutral -- Human and LLM-based Sentiment Annotation of Patient Comments
• Can Large Language Models Serve as Data Analysts? A Multi-Agent Assisted Approach for Qualitative Data Analysis
• The Prompt Report: A Systematic Survey of Prompt Engineering Techniques
• Kimi K2: Open Agentic Intelligence
• LiveBench: A Challenging, Contamination-Limited LLM Benchmark
• Qwen3 Technical Report

The paper

LLMs for Qualitative Data Analysis Fail on Security-specificComments in Human Experiments · Read on arXiv

Maria Camporese, Fabio Massacci, Yuanjun Gong

University of Trento · Vrije Universiteit Amsterdam

[Background:] Thematic analysis of free-text justifications in human experiments provides significant qualitative insights. Yet, it is costly because reliable annotations require multiple domain experts. Large language models (LLMs) seem ideal candidates to replace human annotators. [Problem:] Coding security-specific aspects (code identifiers mentioned, lines-of-code mentioned, security keywords mentioned) may require deeper contextual understanding than sentiment classification. [Objective:] Explore whether LLMs can act as automated annotators for technical security comments by human subjects. [Method:] We prompt four top-performing LLMs on LiveBench to detect nine security-relevant codes in free-text comments by human subjects analyzing vulnerable code snippets. Outputs are compared to human annotators using Cohen's Kappa (chance-corrected accuracy). We test different prompts mimicking annotation best practices, including emerging codes, detailed codebooks with examples, and conflicting examples. [Negative Results:] We observed marked improvements only when using detailed code descriptions; however, these improvements are not uniform across codes and are insufficient to reliably replace a human annotator. [Limitations:] Additional studies with more LLMs and annotation tasks are needed.

DOI: 10.1145/3794763.3798172

Transcript

More episodes

• 2604.03925-AdaptFuse: Training-Free Sequential Preference Inference via Externalized Bayesian Computation
• 2604.18344-DiffTSP: A Conditional Discrete Diffusion Model for Knowledge Graph Triple Set Prediction
• 2512.13399-Dual-Evolving Reinforcement Learning for Automated Reward Design
• 2512.22174-BitFlipScope: Scalable Fault Localization and Recovery for Bit-Flip Corruptions in LLMs
• 2601.07939-SECite: Analyzing and Summarizing Citations in Software Engineering Literature
• 2605.10337-CORTEG: Foundation Models Enable Cross-Modality Representation Transfer from Scalp to Intracranial Brain Recordings
• 2601.22449-Emergence of Physical Intelligence via Controllable Information Production
• 2602.01779-LingLanMedBench: A Unified Benchmark for Evaluating Large Language Models in Traditional Chinese Medicine
• 2602.08353-TKGB: A Comprehensive Benchmark for Temporal Knowledge Graph Forecasting
• 2603.00578-Draft-Thinking: Learning Efficient Reasoning in Long Chain-of-Thought LLMs

← Home