Environment Evolution for Terminal Agents
cs.AI
Submitted: 2026-09-03
Updated: 2026-09-03
License: http://creativecommons.org/licenses/by/4.0/
The gist: Scaling interactive and verifiable environments is critical for training terminal agents.
Terminology
Abstract
Scaling interactive and verifiable environments is critical for training terminal agents. As frontier models become more capable, environments synthesized from scratch become less challenging and thus provide limited learning signals. Recent co-evolution methods iteratively synthesize environments near the model's learnable frontier based on weaknesses exposed during rollouts. However, their dependence on on-policy rollouts limits generalization and the continuous provision of learning signals as the model becomes stronger. In this paper, we propose environment evolution, which incrementally increases environment difficulty off-policy and schedules the evolved environments generation by generation during training to provide continuous learning signals. We derive three evolution directions that influence environment difficulty from the multi-turn learning objective and then implement evolution along these directions through a loop-engineered multi-agent harness. Quantitative rollout experiments with Hy4 preview, Claude Opus 5, and GPT-5.6 Sol show that environment evolution consistently produces more difficult environments. We validate its effectiveness on Qwen3.6-27B and Qwen3.6-35B-A3B through simple long-horizon RL training, improving their performance by 14.4 and 18.0 percentage points on Terminal-Bench 2.1, respectively.
Sources
- What Makes a Good Terminal-Agent Benchmark Task: A Guideline for Adversarial, Difficult, and Legible Evaluation Design
- OpenAI Gym
- Building Effective AI Coding Agents for the Terminal: Scaffolding, Harness, Context Engineering, and Lessons Learned
- SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agent
- Evaluating Large Language Models Trained on Code
- Leveraging Procedural Generation to Benchmark Reinforcement Learning
- Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design
- Toward Scalable Terminal Task Synthesis via Skill Graphs
- AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning
- Endless Terminals: Scaling RL Environments for Terminal Agents
- DRED: Zero-Shot Transfer in Reinforcement Learning via Data-Regularised Environment Design
- GenEnv: Difficulty-Aligned Co-Evolution Between LLM Agents and Environment Simulators
- AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generation
- CLI-Universe: Towards Verifiable Task Synthesis Engine for Terminal Agents
- Prioritized Level Replay
- Meta-Harness: End-to-End Optimization of Model Harnesses
- SPADE: Self-Play in Adaptive Synthetic Executable Environments
- Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers
- Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
- Evolving Curricula with Regret-Based Environment Design
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
- VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection