Agentic Critical Training

The gist The proposed Agentic Critical Training (ACT) is a reinforcement learning paradigm that trains large language models to autonomously develop reasoning about action quality by rewarding correct action selection, leading to genuine self-reflection rather than imitation of pre-constructed reflection text.

Agentic Critical Training

This approach moves beyond imitation learning by training agents to identify the better action among alternatives via reinforcement learning (RL) <ref:2603.08706#pg3>. The core idea is to transform the objective from “imitate the expert action” to “identify the better action,” requiring the model to develop discriminative understanding of action quality <ref:2603.08706#pg5>. ACT pairs each expert action with a model-generated alternative action to form a preference pair at each time step of sequential decision-making <ref:2603.08706#pg6>. The only supervision is whether the model correctly identifies the superior expert action, which drives the model to autonomously develop chain-of-thought (CoT; Wei et al., 2022) reasoning that leads to correct choices <ref:2603.08706#pg6>.

Training Pipeline and Reward Design

The training pipeline consists of two sequential RL stages: Agentic Critical Training followed by RL Action Training, both optimized using Group Relative Policy Optimization (GRPO; Shao et al., 2024) <ref:2603.08706#pg5>. The ACT prompt asks the model to decide which action is better among two candidates, and the model must output the chosen action inside tags <ref:2603.08706#pg19>. The composite reward function is defined as R(s, y) = Racc(a, a+) + Radm(a, Aadmissible) + Rfmt(y) <ref:2603.08706#pg16>. This function includes an accuracy reward for matching the expert action and an admissible action reward for valid but suboptimal actions <ref:2603.08706#pg16>.

Performance Across Benchmarks

ACT consistently improves agent performance across three challenging agent benchmarks (ALFWorld, WebShop, ScienceWorld) when combined with different post-training methods <ref:2603.08706#pg3>. Compared to imitation learning (IL), ACT achieves an average performance gain of 5.07 points, while outperforming reinforcement learning (RL) by 4.62 points <ref:2603.08706#pg9>. Furthermore, ACT demonstrates clear advantages over the Early Experience baseline, yielding an additional 2.42 points improvement on average <ref:2603.08706#pg6>.

Generalization and Reasoning Capabilities

ACT exhibits strong out-of-distribution generalization on agentic benchmarks and improves performance on general reasoning benchmarks (MATH-500 and GPQA-Diamond) without requiring any reasoning-specific training data <ref:2603.08706#pg4>. While IL and Early Experience fail to improve general reasoning, ACT achieves the highest scores on both MATH-500 and GPQA-Diamond despite being trained exclusively on agentic data <ref:2603.08706#pg12>. This is demonstrated by ACT exhibiting self-verification behavior on GPQA-Diamond, where it substitutes back into equations to check consistency <ref:2603.08706#pg12>.

Failure Recovery and Data Transferability

ACT enables failure recovery, as shown in a case study on ALFWorld where the IL model enters an infinite loop repeating a failed action for over 30 steps, whereas the ACT-trained model diagnoses the root cause and issues the correct navigation command <ref:2603.08706#pg10>. Moreover, ACT requires collecting alternative actions from a policy to construct contrastive pairs, but this data can be reused across model sizes for transferability <ref:2603.08706#pg12>. The results validate that ACT’s benefits generalize across model sizes and that the data collection cost can be amortized by reusing data across models of different sizes <ref:2603.08706#pg10>.

Conclusion

ACT trains LLM agents to reason about action quality by contrasting expert and self-generated actions via RL, producing autonomous critical reasoning through RL rather than imitating pre-constructed reflections <ref:2603.08706#pg12>. ACT not only enables strong out-of-distribution generalization on agentic benchmarks but also achieves notable improvements on general reasoning benchmarks without any reasoning-specific training data <ref:2603.08706#pg4>. ACT not only avoids the catastrophic forgetting observed in IL but improves upon the original model, suggesting that agentic RL environments may serve as a viable pathway for enhancing general reasoning capabilities <ref:2603.08706#pg12>. The results indicate that directly training models to evaluate action quality is more effective than supervising them to imitate reflection behaviors <ref:2603.08706#pg6>. ACT also improves OOD generalization, with the gain on top of RL being larger on OOD tasks (3.73pp) than on in-distribution tasks (2.15pp) <ref:2603.08706#pg9>. The paper concludes that ACT is a promising path toward developing more reflective and capable LLM agents <ref:2603.08706#pg2>.

--- Page 1 ---

Agentic Critical Training

Weize Liu, Minghui Liu, Sy-Tuyen Ho, Souradip Chakraborty, Xiyao Wang‡, Furong Huang‡ University of Maryland, College Park ‡Equal advising <ref:2603.08706#pg2>. Training large language models (LLMs) as autonomous agents often begins with imitation learning, but it only teaches agents what to do without understanding why: agents never contrast successful actions against suboptimal alternatives and thus lack awareness of action quality <ref:2603.08706#pg2>. Recent approaches attempt to address this by introducing self-reflection supervision derived from contrasts between expert and alternative actions <ref:2603.08706#pg2>. However, the training paradigm fundamentally remains imitation learning: the model imitates pre-constructed reflection text rather than learning to reason autonomously <ref:2603.08706#pg2>. We propose Agentic Critical Training (ACT), a reinforcement learning paradigm that trains agents to identify the better action among alternatives <ref:2603.08706#pg2>. By rewarding whether the model’s judgment is correct, ACT drives the model to autonomously develop reasoning about action quality, producing genuine self-reflection rather than imitating it <ref:2603.08706#pg2>. Across three challenging agent benchmarks, ACT consistently improves agent performance when combined with different post-training methods <ref:2603.08706#pg2>. It achieves an average improvement of 5.07 points over imitation learning and 4.62 points over reinforcement learning <ref:2603.08706#pg2>. Compared to approaches that inject reflection capability through knowledge distillation, ACT also demonstrates clear advantages, yielding an average improvement of 2.42 points <ref:2603.08706#pg2>. Moreover, ACT enables strong out-ofdistribution generalization on agentic benchmarks and improves performance on general reasoning benchmarks without any reasoning-specific training data <ref:2603.08706#pg2>. These results suggest that ACT is a promising path toward developing more reflective and capable LLM agents <ref:2603.08706#pg2>.

--- Page 2 ---

Agentic Critical Training

(a) Early Experience: Imitated Self-Reflection (b) ACT: Genuine Self-Reflection LLM Generates Reflection Action a∗ is better is better because it moves the agent closer to the target location, while Action a′ fails since the agent is not at the cabinet yet… Expert Action (a∗) Alternative Action (a′) Environment Next State (s∗) Next State (s′) Model Prompt Box Which action is better for the task? Action 1: put cloth in cabinet Action 2: go to cabinet Model Wait… I just cleaned the cloth at the sink. I need to go to the cabinet first before I can put it there! SFT Reflection is imitated from a fixed target string RL Reflection emerges autonomously through RL Figure 1: Comparison of imitated vs. genuine self-reflection. (a) Early Experience executes both actions in the environment, generates a reflection from the resulting states, and trains the model to imitate this fixed text via supervised fine-tuning (SFT) <ref:2603.08706#pg3>. (b) ACT presents two candidate actions and trains the model via RL to select the better one. Since only the selection outcome is rewarded, the model must autonomously develop reasoning about action quality to maximize reward <ref:2603.08706#pg3>.

1. Agentic Critical Training (ACT) drives autonomous reasoning by rewarding action quality judgment, producing genuine self-reflection rather than imitating it. This allows agents to autonomously develop chain-of-thought (CoT; Wei et al., 2022) reasoning that leads to correct choices rather than imitating pre-constructed reflections.

2. The improved system exhibits superior failure recovery by diagnosing root causes, as demonstrated when the IL model enters an infinite loop repeating a failed action for over 30 steps until termination, whereas ACT's model issues the correct navigation command after self-critique.

3. The agent gains strong out-of-distribution generalization, as ACT improves performance on general reasoning benchmarks without requiring reasoning-specific training data, suggesting that learning to evaluate and compare actions can serve as a general mechanism for enhancing reasoning and decision-making abilities in LLM agents.

4. The system overcomes rigid execution errors by internalizing state awareness; instead of following a rigid script (search → click item → select attributes → buy), the ACT agent evaluates candidate actions against the current state, enabling it to back out and search again rather than blindly proceeding.

5. The model preserves and strengthens original reasoning capabilities on general benchmarks like MATH-500, as RL with ACT avoids reasoning collapse because it optimizes for outcome correctness via RL rather than imitating behavioral patterns, successfully avoiding the degradation seen in IL.