Bad Genius: Counterfactual-Guided Harness Evolution Beyond Task-Specific Shortcuts

arXiv:2609.18366 · cs.AI, cs.LG, stat.ML · Submitted 2026-09-16 · Read on arXiv

cs.AI, cs.LG, stat.ML

Submitted: 2026-09-16

Updated: 2026-09-24

Comments: 28 pages, 6 figures; includes references and supplementary material

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Benchmark — 111× in this paper · explained in 2 episodes
• Generalization — 15× in this paper · explained in 13 episodes
• Trace — 8× in this paper · explained in 2 episodes
• Agent Harness — 4× in this paper · explained in 2 episodes
• Core — 4× in this paper · explained in 2 episodes
• Accuracy — 3× in this paper · explained in 1 episode
• Verification — 3× in this paper · explained in 2 episodes
• Agentic AI — 2× in this paper · explained in 1 episode
• Multi-Agent System — 2× in this paper · explained in 6 episodes
• Reward Hacking — 2× in this paper · explained in 8 episodes
• Adaptation — 1× in this paper · explained in 1 episode
• Aggregation — 1× in this paper · explained in 1 episode
• Allocator — 1× in this paper · explained in 1 episode
• Calibration — 1× in this paper · explained in 7 episodes
• Continual Learning — 1× in this paper · explained in 2 episodes
• Failure Modes — 1× in this paper · explained in 3 episodes
• Foundation Model — 1× in this paper · explained in 3 episodes
• Unified Framework — 1× in this paper · explained in 5 episodes

Sources

• EvoSkill: Automated Skill Discovery for Multi-Agent Systems
• SpecRepair: Counter-Example Guided Safety Repair of Deep Neural Networks
• A Robust Optimisation Perspective on Counterexample-Guided Repair of Neural Networks
• A Theory of Reliable Self-Evolution for Agent Harnesses
• How Benchmarks Mis-Score Computer-Use Agents
• StarHarness: Evolving Harnesses with Stratified Search for Enterprise Environments
• METAL: Metamorphic Testing Framework for Analyzing Large-Language Model Qualities
• HarnessEvolve: Learning from Reference Trajectories for Reliable Agent Self-Evolution
• Meta-Harness: End-to-End Optimization of Model Harnesses
• EvoSafeHarness: Evolving Model- and Domain-Specific Harnesses for Securing Agents
• Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses
• Towards Direct Evaluation of Harness Optimizers via Priority Ranking
• OfficeQA Pro: An Enterprise Benchmark for End-to-End Grounded Reasoning
• Proving Test Set Contamination in Black Box Language Models
• Red Teaming Language Models with Language Models
• Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI
• HarnessOpt-Bench: Evaluating LLMs at Harness Optimization

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
• VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection