AuraForge: Scaling Security Supervision for Training Coding Agents

arXiv:2610.00850 · cs.CR, cs.CL, cs.CY · Submitted 2026-10-01 · Read on arXiv

cs.CR, cs.CL, cs.CY

Submitted: 2026-10-01

Updated: 2026-10-01

Code: https://github.com/github/advisory-database

Terminology

• Benchmark — 21× in this paper · explained in 2 episodes
• Verifier — 8× in this paper · explained in 2 episodes
• Core — 6× in this paper · explained in 2 episodes
• Reward Hacking — 6× in this paper · explained in 8 episodes
• Verification — 5× in this paper · explained in 2 episodes
• AI Agents — 3× in this paper · explained in 2 episodes
• Adaptation — 3× in this paper · explained in 1 episode
• Benchmarking — 3× in this paper · explained in 3 episodes
• Consistency — 3× in this paper · explained in 3 episodes
• Fine-Tuning — 3× in this paper · explained in 5 episodes
• Agentic Coding — 2× in this paper · explained in 1 episode
• LLM judge — 2× in this paper · explained in 2 episodes
• Adam — 1× in this paper · explained in 2 episodes
• Generative AI — 1× in this paper · explained in 2 episodes
• KV cache — 1× in this paper · explained in 3 episodes
• Provenance — 1× in this paper · explained in 4 episodes
• Reinforcement Learning — 1× in this paper · explained in 4 episodes
• Trace — 1× in this paper · explained in 2 episodes

Sources

• SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents
• Terminal Wrench: A Dataset of 331 Reward-Hackable Environments and 3,632 Exploit Trajectories
• Vibe Coding in Practice: Motivations, Challenges, and a Future Outlook -- a Grey Literature Review
• ExploitBench: A Capability Ladder Benchmark for LLM Cybersecurity Agents
• Vibe coding: programming through conversation with artificial intelligence
• CyberGym-E2E: Scalable Real-World Benchmark for AI Agents' End-to-End Cybersecurity Capabilities
• Detecting Safety Violations Across Many Agent Traces
• BaxBench: Can LLMs Generate Correct and Secure Backends?
• ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?
• PATCHEVAL: A New Benchmark for Evaluating LLMs on Patching Real-World Vulnerabilities
• SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
• SWE-smith: Scaling Data for Software Engineering Agents

Related papers

• SoK: AI-Augmented Binary Reversing
• Relaxed Sender Anonymity for CBDC Interbank Settlement: A Zero-Knowledge Approach on Permissioned EVM
• Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
• Efficient Fuzzy PSI under One-Sided Assumptions
• Sealing the Audit-Runtime Gap for LLM Skills
• Token Composition: A Graph Based on EVM Logs