Compared to What? A Human-Anchored Security Benchmark for LLM-Generated Infrastructure-as-Code

arXiv:2608.28021 · cs.CR, cs.AI, cs.MA, cs.SE · Submitted 2026-08-28 · Read on arXiv

cs.CR, cs.AI, cs.MA, cs.SE

Submitted: 2026-08-28

Updated: 2026-09-21

Code: https://github.com/AnimeshShaw/GenIaC-SecBench

Terminology

• Benchmark — 16× in this paper · explained in 2 episodes
• LLM judge — 6× in this paper · explained in 2 episodes
• Hallucination — 5× in this paper · explained in 3 episodes
• Calibration — 3× in this paper · explained in 7 episodes
• Consistency — 3× in this paper · explained in 3 episodes
• LLM-as-a-Judge — 3× in this paper · explained in 3 episodes
• Verification — 3× in this paper · explained in 2 episodes
• Chain-of-Thought Prompting — 2× in this paper · explained in 2 episodes
• Pairwise Comparisons — 2× in this paper · explained in 2 episodes
• Alignment — 1× in this paper · explained in 1 episode
• Commentary — 1× in this paper · explained in 2 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• Hallucination Detection — 1× in this paper · explained in 2 episodes
• Longitudinal Data — 1× in this paper · explained in 2 episodes
• Reinforcement Learning — 1× in this paper · explained in 4 episodes

Sources

• Security-First Evaluation of Text-to-Terraform: Benchmarking LLMs and SLMs for Secure IaC Generation
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• A Survey on LLM-as-a-Judge
• Empirical Standards for Software Engineering Research

Related papers

• SoK: AI-Augmented Binary Reversing
• Relaxed Sender Anonymity for CBDC Interbank Settlement: A Zero-Knowledge Approach on Permissioned EVM
• Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
• Efficient Fuzzy PSI under One-Sided Assumptions
• Sealing the Audit-Runtime Gap for LLM Skills
• Token Composition: A Graph Based on EVM Logs