On the Fragility of Self-Improving Agents: Variance, Task Order, and Underspecification
cs.AI, cs.CL, cs.LG
Submitted: 2026-08-18
Updated: 2026-09-06
Comments: Code: https://github.com/SalesforceAIResearch/self-improve-fragility Data: https://huggingface.co/datasets/Salesforce/self-improve-fragility
Code: https://github.com/SalesforceAIResearch/self-improve-fragility
License: http://creativecommons.org/licenses/by/4.0/
The gist: Memory-based self-improving agents--those that learn from an online stream of tasks and improve over time by maintaining a textual memory bank--have shown great promise in recent literature.
Terminology
Abstract
Memory-based self-improving agents--those that learn from an online stream of tasks and improve over time by maintaining a textual memory bank--have shown great promise in recent literature. However, the reliability aspects of these methods have been critically overlooked. In this work, we conduct a comprehensive re-evaluation of two memory-based methods, broadening the scope of evaluation along two axes: (1) including multiple self-improving runs to quantify variance, and (2) shuffling the tasks to investigate the effect of task order. Through these experiments, we make two observations that expose the fragility of current methods: First, agent evaluation is inherently noisy in complex environments and on multi-step tasks, and stacking a self-improving loop on top can further amplify this noise. Second, the agent's improvement is highly dependent on task order. Prior works often adopt default orderings that impose an implicit curriculum, acting as a hidden prerequisite for success. To better understand this fragility, we manually examine the agents' memory and hypothesize that task and environment underspecification contribute to this fragility. We validate this hypothesis by incorporating information that enables better specification, such as detailed rubrics and environment feedback, into the memory construction process. While this added information partially closes the performance degradation in previous experiments, significant gaps still remain, suggesting that other uncharacterized factors contribute to this fragility. Looking ahead, our work advocates for more rigorous evaluation protocols for self-improving agents by reporting results across multiple runs and stress-testing them under challenging conditions. Moreover, our findings on underspecification call for systems and interfaces that enable effective human oversight, preventing agents from failing in unforeseeable ways.
Sources
- Goedel Machines: Self-Referential Universal Problem Solvers Making Provably Optimal Self-Improvements
- NNetNav: Unsupervised Learning of Browser Agents Through Environment Interaction in the Wild
- Towards a Science of AI Agent Reliability
- ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory
- SCUBA: Salesforce Computer Use Benchmark
- $\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
- Learning Next Action Predictors from Human-Computer Interaction
- MemGPT: Towards LLMs as Operating Systems
- Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory
- SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks
- Skills-Coach: A Self-Evolving Skill Optimizer via Training-Free GRPO
- Continual Learning via Sparse Memory Finetuning
- Toolformer: Language Models Can Teach Themselves to Use Tools
- GDPval: Evaluating AI Model Performance on Real-World Economically Valuable Tasks
- SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
- EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings
- Establishing Best Practices for Building Rigorous Agentic Benchmarks
- Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation
- Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation
- Monitoring Monitorability
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
- VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection