KaliBench: A Fine-Grained Benchmark for Cybersecurity Tool Use on Kali Linux with Runtime-Free Verifiable Rewards

arXiv:2610.02206 · cs.CL, cs.AI, cs.CR · Submitted 2026-10-01 · Read on arXiv

cs.CL, cs.AI, cs.CR

Submitted: 2026-10-01

Updated: 2026-10-01

Code: https://github.com/RISys-Lab/KaliBench

Project page: https://risys-lab.github.io/KaliBench

Terminology

• Benchmark — 48× in this paper · explained in 2 episodes
• GRPO — 37× in this paper · explained in 6 episodes
• Verification — 29× in this paper · explained in 2 episodes
• Total Score — 23× in this paper
• Exact Correct — 13× in this paper
• Accuracy — 10× in this paper · explained in 1 episode
• Verifiable Rewards — 10× in this paper · explained in 2 episodes
• Core — 9× in this paper · explained in 2 episodes
• Fine-Tuning — 8× in this paper · explained in 5 episodes
• Reinforcement Learning — 7× in this paper · explained in 4 episodes
• Human-in-the-Loop — 6× in this paper · explained in 3 episodes
• Verifier — 6× in this paper · explained in 2 episodes
• Hallucination — 5× in this paper · explained in 3 episodes
• Supervised Fine-Tuning — 5× in this paper
• Agentic Evaluation — 4× in this paper · explained in 1 episode
• Benchmarking — 4× in this paper · explained in 3 episodes
• Tool Accuracy — 4× in this paper
• Alignment — 3× in this paper · explained in 1 episode
• Reinforcement Learning with Verifiable Rewards — 3× in this paper
• Robustness — 3× in this paper · explained in 5 episodes
• Adam — 2× in this paper · explained in 2 episodes
• Evaluation Dimensions — 2× in this paper · explained in 3 episodes
• Grounding — 2× in this paper · explained in 3 episodes
• Institutional Review Board — 2× in this paper
• Abstraction — 1× in this paper · explained in 1 episode
• AdamW Optimizer — 1× in this paper · explained in 1 episode
• Adaptation — 1× in this paper · explained in 1 episode
• Agentic Task — 1× in this paper · explained in 1 episode
• Agentic Tasks — 1× in this paper · explained in 1 episode
• Aggregation — 1× in this paper · explained in 1 episode
• Clustering — 1× in this paper · explained in 2 episodes
• Commentary — 1× in this paper · explained in 2 episodes
• Cosine Similarity — 1× in this paper · explained in 3 episodes
• Failure Modes — 1× in this paper · explained in 3 episodes
• Foundation Model — 1× in this paper · explained in 3 episodes
• Foundation Models — 1× in this paper · explained in 7 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• Generative Model — 1× in this paper · explained in 2 episodes
• Generative Models — 1× in this paper · explained in 2 episodes
• Group Relative Policy Optimization (GRPO) — 1× in this paper · explained in 5 episodes
• Initial Data — 1× in this paper · explained in 2 episodes
• KU Research Center for Advanced Intelligent Systems — 1× in this paper
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
• LoRA Adapters — 1× in this paper · explained in 3 episodes
• Parameter-Efficient Fine-Tuning — 1× in this paper · explained in 2 episodes
• RLVR with Group Relative Policy Optimization — 1× in this paper
• Reward Function — 1× in this paper · explained in 2 episodes
• Semantic Alignment — 1× in this paper · explained in 3 episodes
• Structured Prompting — 1× in this paper · explained in 2 episodes
• Transformer — 1× in this paper · explained in 2 episodes
• alpha — 1× in this paper · explained in 1 episode

Sources

• Qwen3-Coder-Next Technical Report
• DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
• GLM-5: from Vibe Coding to Agentic Engineering
• The Llama 3 Herd of Models
• LoRA: Low-Rank Adaptation of Large Language Models
• SecBench: A Comprehensive Multi-Dimensional Benchmarking Dataset for LLMs in Cybersecurity
• gpt-oss-120b & gpt-oss-20b Model Card
• Qwen2.5 Technical Report
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
• Gemma 3 Technical Report
• Llama-3.1-FoundationAI-SecurityLLM-8B-Instruct Technical Report
• Qwen3 Technical Report
• Llama-3.1-FoundationAI-SecurityLLM-Reasoning-8B Technical Report
• CS-Eval: A Comprehensive Large Language Model Benchmark for CyberSecurity
• DefenderBench: A Toolkit for Evaluating Language Agents in Cybersecurity Environments

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering