Benchmark Scores Are Pipeline-Dependent: A Reliability Audit of Cybersecurity LLM Benchmarks

arXiv:2609.08765 · cs.CR, cs.AI, cs.CL · Submitted 2026-09-08 · Read on arXiv

cs.CR, cs.AI, cs.CL

Submitted: 2026-09-08

Updated: 2026-09-08

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Benchmark — 8× in this paper · explained in 2 episodes
• Core — 1× in this paper · explained in 2 episodes
• Failure Modes — 1× in this paper · explained in 3 episodes
• Large Language Model (LLM) — 1× in this paper · explained in 3 episodes

Related papers

• SoK: AI-Augmented Binary Reversing
• Relaxed Sender Anonymity for CBDC Interbank Settlement: A Zero-Knowledge Approach on Permissioned EVM
• Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
• Efficient Fuzzy PSI under One-Sided Assumptions
• Sealing the Audit-Runtime Gap for LLM Skills
• Token Composition: A Graph Based on EVM Logs