Evolving Excellence: Automated Optimization of LLM-based Agents

arXiv:2512.09108 · cs.SE, cs.AI · Submitted 2025-12-09 · Read on arXiv

cs.SE, cs.AI

Submitted: 2025-12-09

Updated: 2026-09-03

Code: https://github.com/pppyb/mini-swe-agent

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Benchmark — 53× in this paper · explained in 2 episodes
• Accuracy — 39× in this paper · explained in 1 episode
• Prompt Engineering — 9× in this paper · explained in 4 episodes
• Large Language Model — 8× in this paper
• Verification — 5× in this paper · explained in 2 episodes
• Generalization — 4× in this paper · explained in 13 episodes
• Benchmarking — 3× in this paper · explained in 3 episodes
• Multi-Agent System — 3× in this paper · explained in 6 episodes
• Agentic AI — 2× in this paper · explained in 1 episode
• Agentic AI systems — 2× in this paper · explained in 1 episode
• Agentic System — 2× in this paper · explained in 2 episodes
• Algorithmic Improvements — 2× in this paper · explained in 1 episode
• Bayesian Optimization — 2× in this paper · explained in 3 episodes
• Failure Modes — 2× in this paper · explained in 3 episodes
• Large Language Model (LLM) — 2× in this paper · explained in 3 episodes
• Large Language Models (LLMs) — 2× in this paper · explained in 6 episodes
• Local Optimization — 2× in this paper · explained in 2 episodes
• Multi-objective Optimization — 2× in this paper · explained in 4 episodes
• Reinforcement Learning — 2× in this paper · explained in 4 episodes
• Semantic Understanding — 2× in this paper · explained in 2 episodes
• Trace — 2× in this paper · explained in 2 episodes
• Abstraction — 1× in this paper · explained in 1 episode
• Adam — 1× in this paper · explained in 2 episodes
• Agent Architectures — 1× in this paper · explained in 1 episode
• Aggregation — 1× in this paper · explained in 1 episode
• Core — 1× in this paper · explained in 2 episodes
• Failure Taxonomy — 1× in this paper · explained in 2 episodes
• Generalizability — 1× in this paper · explained in 6 episodes
• Generative AI — 1× in this paper · explained in 2 episodes
• Human-in-the-Loop — 1× in this paper · explained in 3 episodes
• Iterative Refinement — 1× in this paper · explained in 3 episodes
• LLM judge — 1× in this paper · explained in 2 episodes
• Monte Carlo Tree Search — 1× in this paper
• Objective Function — 1× in this paper · explained in 2 episodes
• Reasoning Traces — 1× in this paper · explained in 3 episodes
• Sample Efficiency — 1× in this paper · explained in 4 episodes
• Transfer Learning — 1× in this paper · explained in 10 episodes
• Verifier — 1× in this paper · explained in 2 episodes

Sources

• GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning
• AFlow: Automating Agentic Workflow Generation
• Why Do Multi-Agent LLM Systems Fail?
• Training Verifiers to Solve Math Word Problems
• Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution
• Language Models for Code Optimization: Survey, Challenges and Future Directions
• Automated Design of Agentic Systems
• ALE-Bench: A Benchmark for Long-Horizon Objective-Driven Algorithm Engineering
• ShinkaEvolve: Towards Open-Ended And Sample-Efficient Program Evolution
• AlphaEvolve: A coding agent for scientific and algorithmic discovery
• Towards Scientific Intelligence: A Survey of LLM-based Scientific Agents
• Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineering
• ReAct: Synergizing Reasoning and Acting in Language Models
• Large Language Models Are Human-Level Prompt Engineers

Related papers

• Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
• GitSkills: A Dataset of Agent Skills on GitHub
• SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces
• PackMonitor: Enabling Zero Package Hallucinations Through Decoding-Time Monitoring
• IntentCoding: Amplifying User Intent in Code Generation
• Incentives and Outcomes in Bug Bounties