SecProbe: Adaptive Evaluation of Coding Agents on Cybersecurity Vulnerabilities

arXiv:2609.33763 · cs.CR, cs.AI · Submitted 2026-09-27 · Read on arXiv

cs.CR, cs.AI

Submitted: 2026-09-27

Updated: 2026-09-27

Terminology

• Benchmark — 16× in this paper · explained in 2 episodes
• Benchmarking — 12× in this paper · explained in 3 episodes
• Alignment — 6× in this paper · explained in 1 episode
• AI Agents — 5× in this paper · explained in 2 episodes
• Calibration — 4× in this paper · explained in 7 episodes
• Cosine Similarity — 4× in this paper · explained in 3 episodes
• Adaptive Selection — 3× in this paper · explained in 1 episode
• Clustering — 2× in this paper · explained in 2 episodes
• Core — 2× in this paper · explained in 2 episodes
• Failure Modes — 2× in this paper · explained in 3 episodes
• Robustness — 2× in this paper · explained in 5 episodes
• Synthetic Data Generation — 2× in this paper · explained in 5 episodes
• Verification — 2× in this paper · explained in 2 episodes
• Accuracy — 1× in this paper · explained in 1 episode
• Agentic Task — 1× in this paper · explained in 1 episode
• Agentic Tasks — 1× in this paper · explained in 1 episode
• Aggregation — 1× in this paper · explained in 1 episode
• Analyzer — 1× in this paper · explained in 1 episode
• Common Weakness Enumeration — 1× in this paper
• Consistency — 1× in this paper · explained in 3 episodes
• Data Augmentation — 1× in this paper · explained in 7 episodes
• Fine-Tuning — 1× in this paper · explained in 5 episodes
• Fisher Information — 1× in this paper · explained in 3 episodes
• Foundation Model — 1× in this paper · explained in 3 episodes
• Foundation Models — 1× in this paper · explained in 7 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• Generative AI — 1× in this paper · explained in 2 episodes
• Information Leakage — 1× in this paper · explained in 2 episodes
• Instruction Tuning — 1× in this paper · explained in 2 episodes
• Item Response Theory — 1× in this paper
• Large Language Model (LLM) — 1× in this paper · explained in 3 episodes
• Matrix Completion — 1× in this paper · explained in 4 episodes
• Objective Function — 1× in this paper · explained in 2 episodes
• Sycophancy — 1× in this paper · explained in 4 episodes
• Trace — 1× in this paper · explained in 2 episodes

Sources

• RAS-Eval: A Comprehensive Benchmark for Security Evaluation of LLM Agents in Real-World Environments
• Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models
• A Rosetta Stone for AI Benchmarks
• MiniMax Sparse Attention
• UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
• SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents
• How Safe Are AI-Generated Patches? A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-bench
• The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark
• Kimi K3: Open Frontier Intelligence
• SWE-Mirror: Scaling Issue-Resolving Datasets by Mirroring Issues Across Repositories
• Simple synthetic data reduces sycophancy in large language models
• How Predictable Are Large Language Model Capabilities? A Case Study on BIG-bench
• Oasis: One Image is All You Need for Multimodal Instruction Data Synthesis
• SWE-Explore: Benchmarking How Coding Agents Explore Repositories
• Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models

Related papers

• SoK: AI-Augmented Binary Reversing
• Relaxed Sender Anonymity for CBDC Interbank Settlement: A Zero-Knowledge Approach on Permissioned EVM
• Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
• Efficient Fuzzy PSI under One-Sided Assumptions
• Sealing the Audit-Runtime Gap for LLM Skills
• Token Composition: A Graph Based on EVM Logs