PhantomEnvironments: Training LLM Agents in Fictional Worlds

Training LLM agents with reinforcement learning (RL) is bottlenecked by environments, which must provide verifiable rewards, support long-horizon interaction, and scale cheaply. The gist: LLMs can instead be trained into capable search agents using synthetic environments generated entirely by rules, whose generation requires no LLM and has zero marginal cost.

PhantomEnvironments Construction

The core innovation is the creation of PhantomEnvironments, multi-turn RL environments derived from fictional worlds and template-generated multi-hop questions. These environments are built entirely from codified rules, requiring no humans or LLMs in the pipeline, which results in them being verifiable by construction and incur zero generation cost. The worlds consist of fictional entities connected through programmed relationships (like family or friendship), and articles about these individuals are generated using template-based formats. Questions are generated with context-free grammars, ensuring every question-answer pair is fully verifiable by executing a parallel Prolog query that returns ground-truth answers.

Training Methodology

The paper follows the RL fine-tuning recipe of Search-R1, instructing models to use XML tags like for queries and for final answers. The optimization uses Group Relative Policy Optimization (GRPO), where the model is rewarded only on the final answer, and environment outputs within tags are masked out of the objective function. Training involves approximately 55K questions in these synthetic environments, which are compared against a random subsample of 55K questions from the NaturalQuestions-HotpotQA corpus to assess transfer to real-world data.

Key Findings on Transfer and Scaling

The agents trained in PhantomEnvironments demonstrate significant and consistent transfer to real-world multi-hop search benchmarks. Performance improves by roughly 1.7× on older benchmarks (like HotpotQA) and 2.2× on newer, harder ones (like SynthWorlds-RM). Crucially, the paper shows that Qwen2.5 models learn to allocate their search budget roughly linearly with question difficulty—an emergent property termed search scaling—suggesting this is a skill learned from environment interaction alone. Furthermore, agents trained in these synthetic environments are robust to larger-scale deployments with noisy search results and are effective at closing the Knowledge Advantage (KA) gap when evaluation targets fall beyond LLM training cutoffs.

Complexity Axes Analysis

The study analyzed three orthogonal axes of complexity: linear hops over entities, comparisons of attributes, and constraints that filter candidates. The results show that Hops is the strongest axis that drives real-world transfer, as linear hops are at least as effective as the other two axes combined. Specifically, comparison questions target matching real-world categories and significantly outperform constraint questions, which are shown to hurt transfer because they reward a verbatim-query shortcut rather than decomposition.

Generalization and Robustness

The agents trained on fixed fictional universes generalize well to unseen environments of the same size and even 10× larger universes, confirming that they have learned generalizable agentic search skill rather than memorizing facts. The analysis further reveals that learning agentic search through RL largely teaches a procedural skill, which can be learned complement to the LLMs’ memorized parametric knowledge. The paper concludes that rule-generated synthetic environments are a zeromarginal-cost and effective source of agent training data.

Limitations and Future Directions

The limitations noted include the lag on in-domain benchmarks where knowledge memorization helps, and the necessity to design environments to avoid reward shortcuts. Future work is suggested for environment composition, investigating if one can read off a recipe of synthetic axes that closes specific capability gaps. The authors also note that compute budgets limit training larger models and open-web benchmarks like BrowseComp-Plus.

AI Use Statement

In this work, we used generative AI tools for code implementation and experimentation. We have not used generative AI tools for generating synthetic datasets (the paper is about LLM-free synthetic data), or interpreting results; the rest of the required disclosure tasks are not applicable to this work. Additionally, we used generative AI tools for brainstorming, editing for readability, and plotting. We have reviewed all AI-assisted work by verifying code and confirming all literature. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI.

Reproducibility Statement

We use open-source LLMs, training code, and evaluation benchmarks, and our training experiments are across multiple seeds. All details are in the appendix. We report standard errors in all tables and plots and note significance. We have open-sourced our code at github.com/kilian-group/phantom-envs.

Acknowledgements

Authors acknowledge help from AI models in various project phases. DG is supported by Empire AI Postdoctoral Fellowship.

Here are specific improvements for AI systems based on the PHANTOMENVIRONMENTS: TRAINING LLM AGENTS IN FICTIONAL WORLDS paper, categorized by capability.

) 1. Develop Zero-Marginal-Cost, Verifiable Synthetic RL Training Environments.

The system can generate entire multi-turn reinforcement learning trajectories from fictional worlds using only codified rules (e.g., Prolog queries), requiring no LLM generation or human curation for the environment itself.

2. Enhance Agentic Search Skills via Rule-Generated Transfer Learning.

The improved AI system can be fine-tuned using Reinforcement Learning (RL) on these synthetic environments to master core agentic search skills, specifically:

• Decomposing complex multi-hop questions into sub-questions.

• Formulating targeted, focused search queries (tags).

• Retrieving information from a knowledge corpus.

• Composing knowledge across multiple turns of interaction.

3. Achieve Robust Transfer to Real Multi-Hop Benchmarks.

By training on rule-generated synthetic data, the resulting LLM agents can achieve significant performance gains (up to 7.1×) on real-world multi-hop search benchmarks (e.g., SynthWorlds, FRAMES). This means the agent learns generalized search strategy rather than memorizing specific facts from a fixed training corpus.

4. Implement Emergent Search Scaling Behavior.

The system can be designed such that the model learns to allocate its search budget roughly linearly with question difficulty, demonstrating search scaling. This allows for predictable performance gains as questions become more complex, regardless of the underlying model size (emergent in capable LLMs like Qwen2.5).

5. Optimize Environment Complexity for Targeted Skill Acquisition.

The system can dynamically compose training environments along specific complexity axes:

• Focus on pure linear hops to drive general transfer.

• Use comparison questions to improve performance on real-world comparison tasks (e.g., matching attributes).

• Avoid constraint questions that reward verbatim query shortcuts, which actively hinder agentic decomposition.

6. Ensure Robustness to Noisy and Large-Scale Deployment.

The agents trained in these synthetic environments are robust to deployment challenges, including:

• Overlapping or near-duplicate documents in a large pooled search index (simulating noisy search).

• Out-of-domain evaluation benchmarks (where knowledge memorization is less useful than learned skills), as the training decoupled the agent from specific temporal data snapshots.

Training LLM agents with reinforcement learning (RL) is bottlenecked by environments, which must provide verifiable rewards, support long-horizon interaction, and scale cheaply. Existing approaches rely on costly human-curated data or on LLM-generated environments that risk hallucinations and benchmark contamination. We show that LLMs can instead be trained into capable search agents using synthetic environments generated entirely by rules, whose generation requires no LLM and has zero marginal cost. We build PhantomEnvironments, multi-turn RL environments from fictional worlds, where agents must search a corpus of templated articles to answer multi-hop questions. Despite sharing no facts with the real world, these strikingly simple environments yield agents that transfer to real-world multi-hop search benchmarks, often outperforming real-world training data on newer benchmarks. Trained agents generalize to unseen fictional universes, and Qwen models learn to scale their search budget roughly linearly with question difficulty, suggesting emergent search scaling from environment interaction alone. Ablating environment complexity reveals that hop count drives transfer more than constraints or comparisons: even the simplest rule-generated environments are a surprisingly effective, free resource for training generalizable LLM agents.