Meta-TTL: Meta-Learning Self-Improvement Policies for Language Agents

Test-Time Learning (TTL) enables language agents to iteratively refine their performance through repeated interactions at inference time, and this paper introduces META-TTL, a framework that formulates discovery of effective adaptation policies as a bi-level optimization problem. This approach learns an adaptation policy from task environments by optimizing it for downstream improvement at test time, suggesting that optimal adaptation policies should be learned from task environments rather than being hand-engineered based on human intuition.

How it works

The framework is structured around a bi-level optimization problem consisting of an inner loop and an outer meta-training loop. The inner TTL loop executes the standard TTL process where an LLM agent interacts with the environment over a series of episodes and adapts based on prior attempts, measuring how well a candidate adaptation policy helps the agent correct errors across sequential episodes. Guided by this performance, the outer loop employs evolutionary search over a diverse distribution of training tasks to continually optimize the adaptation policy.

Learned Adaptation Policies for Language Agents

The core mechanism involves two distinct policies: an actor policy that determines behavior within a single episode, and an adaptation policy that updates the actor policy based on accumulated experience. The goal is to learn this adaptation policy, which maps past experience to future behavioral improvement, rather than relying on fixed or hand-crafted rules. This learning is instantiated in prompt space, where the actor's behavior is mediated through system prompt rewriting rather than parameter updates. Specifically:

1. The actor policy determines behavior within an episode: The actor policy π determines behavior within a single episode, selecting actions given the current observation.

2. The adaptation policy operates at a higher level: the adaptation policy f operates at a higher level: after each episode, it observes the accumulated experience and produces an updated actor policy for the next attempt.

3. The learnable component is the meta-prompt ϕ, which fully specifies the adaptation policy: The meta-prompt ϕ fully specifies the adaptation policy: it determines what aspects of past experience the meta-agent attends to, how it diagnoses failures, and what form of guidance it produces.

Reflective Meta-Training

The objective of meta-training is to find a meta-prompt ϕ∗ that maximizes expected TTL performance on the training tasks: ϕ∗ = argmax ϕ Eg∼Dtrain W-AUC(ξϕg). This is achieved through reflective prompt evolution, similar to GEPA, where candidate meta-prompts are proposed through reflection and selected by session-level W-AUC. The process involves:

1. Proposal and Local Validation: A parent meta-prompt is sampled, run on a training task (TTL session), and a candidate is proposed based on reflection. This candidate undergoes local validation on the same task to ensure improvement before proceeding to global validation.

2. Global Validation: Candidates that pass local validation are evaluated across all validation tasks in Dval, and the pool of experts is updated if a new best score is achieved for any task.

3. Expert Selection: After training, a single optimized meta-prompt ϕ∗ is selected from the expert pool by choosing the expert with the highest average validation score, often using per-game z-score normalization to select more uniformly strong candidates.

Emergent Adaptation Policies

The optimized meta-prompt (ϕ∗) exhibits several qualitatively distinct features that emerge through evolutionary optimization:

1. Mandatory structured output: The meta-prompt specifies six required output sections, including diagnosis of what happened, durable game facts, and a recommended route with save points. This structure forces the meta-agent to separate diagnosis, fact extraction, planning, and scripting.

2. Explicit credit assignment protocol: The meta-prompt requires itemizing which actions scored points and how to reproduce them, which actions caused death or threats, which wasted turns (dead ends), and which blocked progress (locked doors).

3. Grounded fact accumulation: A Game facts to remember section must record map links, required triggers, working command syntax, and non-working verbs the parser rejected. This is constrained to be evidenced by the most recent episode log to prevent hallucination.

4. Exploration management: The meta-prompt enforces a disciplined exploration policy: at most one new experiment per episode, always under a save/restore point, with an explicit fallback if two attempts at the same approach fail.

5. Concrete action scripts: The meta-prompt requires a 15–25 command opening script that reproduces known scoring actions quickly before attempting new objectives.

6. Conditional fact banks: Game-specific knowledge is included but activated only when the game identity is confirmed from the episode log via a CRITICAL ADAPTATION RULE ensuring irrelevant fact banks are ignored.

Evaluation

META-TTL was evaluated on Jericho, WebArena-Lite, and τ2-bench across both in-distribution (ID) and out-of-distribution (OOD) settings.

Here are specific, actionable improvements for AI systems based on the META-TTL framework:

1. Improve on-the-fly performance in novel or unfamiliar environments by implementing a bi-level meta-learning loop that learns an adaptation policy (encoded as a natural language meta-prompt) rather than relying on fixed, hand-crafted heuristics.

2. Enable language agents to achieve sustained, iterative self-improvement across sequential episodes during inference time by dynamically updating their behavioral adaptation policy based on accumulated experience from previous failures and successes.

3. Develop a Meta-Agent capable of acting as a learned learning algorithm that maps past trajectories (experience history) to future behavioral improvement by optimizing its system prompt against a distribution of training tasks.

4. Create agents that demonstrate strong generalization capabilities to out-of-distribution (OOD) tasks, where the learned adaptation policy encodes transferable strategies that allow them to solve unseen environments effectively, as evidenced by gains on benchmarks like Jericho and τ2-bench OOD settings.

5. Implement a robust Expert Selection mechanism using per-game z-score normalization during meta-training to ensure the agent learns task-agnostic adaptation strategies rather than overfitting to easily improved or single tasks, leading to more reliable generalization.

6. Produce highly structured and actionable feedback by forcing the adaptation policy (meta-prompt) to follow a mandatory output format that explicitly includes diagnosis, durable fact extraction, next-episode priorities, and concrete command scripts tailored for the current game context.

7. Enhance agent planning by requiring the learned policy to enforce disciplined exploration management (e.g., one new branch per episode under save/restore points) and explicit credit assignment protocols that itemize which specific actions caused progress versus those that were wasted turns or dead ends.

8. Create an optimization strategy where the adaptation mechanism itself is treated as a learnable object, allowing researchers to compare the efficacy of optimizing the adaptation policy versus optimizing the primary task-solving actor, demonstrating superior gains in W-AUC and overall performance across diverse benchmarks.

Test-Time Learning (TTL) enables language agents to iteratively refine their performance through repeated interactions with the environment at inference time. At the core of TTL is a self-improvement policy that updates the actor policy based on experience from previous episodes, thereby improving future behavior. Existing methods rely on hand-crafted self-improvement rather than optimizing them for downstream improvement. We argue that optimal self-improvement policies should be learned from task environments, not hand-engineered based on human intuition. To achieve this, we introduce Meta-TTL, a framework that formulates the discovery of effective self-improvement policies as a bi-level optimization problem. Within this framework, the inner loop executes the standard TTL process, measuring how effectively a candidate self-improvement policy helps an agent correct errors across sequential episodes. Guided by the agent's performance, the outer loop performs reflective meta-training across diverse training tasks, using a balanced improvement score (BIS) to balance task contributions during candidate selection. We evaluate Meta-TTL on Jericho, WebArena-Lite, and-bench across both in-distribution (ID) and out-of-distribution (OOD) settings. Meta-TTL consistently outperforms existing baselines, improving TTL over the strongest baseline by up to 23% on ID tasks and 27% on OOD tasks. These results suggest that the optimized self-improvement policy encodes transferable meta-strategies that generalize beyond the training task distribution.