EDGEGEN: Improving Tool-Calling Agents Beyond Happy Paths with Synthetic Edge Case Generation
cs.AI
Submitted: 2026-09-21
Updated: 2026-09-21
Comments: NA
License: http://creativecommons.org/licenses/by/4.0/
The gist: Tool-calling LLM agents are increasingly deployed in enterprise applications.
Terminology
Abstract
Tool-calling LLM agents are increasingly deployed in enterprise applications. However, effective evaluation and optimization require high-quality, diverse task datasets that are often difficult to obtain due to privacy and other constraints. Existing synthetic task generation methods often produce generic tasks that ignore an agent's underlying state or database and fail to reflect real-world usage diversity. We propose EdgeGen, a synthetic task generation framework that extracts compliance rules from an agent's specification and uses them to generate database-grounded edge-case tasks designed to violate these rules. When combined with existing synthetic data generation techniques, EdgeGen enables agent improvement through finetuning and harness optimization. The resulting pipeline forms a fully automated closed-loop system that requires no human annotation. Finetuning on data generated by EdgeGen yields a consistent mean progress improvement of 2 percent to 42 percent on tau2bench airline domain, while other baseline methods show degradation for some models. On the other hand, for harness optimization, our method shows a mean progress improvement of 10 percent and 30 percent over the human-curated and base harnesses, respectively, for the Gemma-4-e4b model.
Sources
- Reinforcement Learning for Long-Horizon Interactive LLM Agents
- Meta-Harness: End-to-End Optimization of Model Harnesses
- Best Practices and Lessons Learned on Synthetic Data
- $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment
- AgentBench: Evaluating LLMs as Agents
- On the Diversity of Synthetic Data and its Impact on Training Large Language Models
- Qwen2 Technical Report
- $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
- Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents
- Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models
- Agent-SafetyBench: Evaluating the Safety of LLM Agents
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
- VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection