Retrospective Progress-Aware Self-Refinement for LLM Agent Training

arXiv:2606.14302 · cs.CL · Submitted 2026-06-12 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "Retrospective Progress-Aware Self-Refinement for LLM Agent Training".

Tom: LLM-based agents trained with reinforcement learning optimize step-wise action prediction but lack metacognitive awareness of task progress, inducing a gap that hinders longhorizon scaling.

Jane: First, who's behind it and why it matters.

Title and authors: Tom: So, this paper is talking about how we can fix a big problem with agents that are supposed to do long tasks. Basically, these AI agents are good at predicting the next step, but they don't actually know how much progress they've made on the whole task.

Jane: Exactly. They lack that metacognition—that awareness of their own progress—and it really messes up scaling them to longer jobs. This paper introduces a framework called RePro, which is designed to train agents to generate these progress signals by using a specific way of rolling out actions and then looking back at the results.

Lu: The core idea here is moving away from just relying on external supervision or simple reward models for this kind of self-assessment. They show that you can teach an agent to assess its own progress from the outcomes of completed tasks, rather than just prompting it online.

Meng: That sounds really cool conceptually, but I gotta ask—how do they actually make sure this learning sticks? Is it just a fancy way to get better step-by-step predictions without solving the bigger awareness issue?

Tom: Right, that’s the million-dollar question. The paper suggests that progress awareness needs dedicated training, because online prompting doesn't work well for performance, but seeing a completed trajectory helps. This RePro framework is built on a forward-then-reflect rollout paradigm.

Jane: That forward-then-reflect part is key to understanding the whole setup: the agent first executes actions while trying to guess its progress, and then after it finishes, it goes back and reassesses its progress based on the final outcome.

Lu: The framework has two main stages for teaching this reflection format. First, there's a Retrospection Warmup where they teach the agent the correct format from some simple external demonstrations so it knows how to reflect later.

Meng: And then there’s RePro-PO, which uses that retrospective progress signal to create a composite reward. So instead of just one sparse outcome reward, they have several per-step signals that guide the learning process.

Title and authors: Tom: That's what makes the difference in training—these composite rewards include things like retrospective progress shaping and onlineretrospective alignment, which are designed to complement that sparse final result.

Jane: The paper shows that this method actually helps agents develop metacognitive awareness, where their progress estimates can distinguish between successful and failed attempts much better than before.

Lu: They tested this across different models and environments, specifically on WebShop, ALFWorld, and Sokoban. The results show consistent performance gains when using RePro compared to just the baseline training methods.

Meng: What about the specific numbers they found? I'm interested in seeing how much better these agents actually perform when we look at those benchmarks.

Tom: On WebShop, they report improvements in absolute task success rate of plus eight point nine eight percent for one model size, plus eleven point five seven percent for another, and plus five point eight two percent for a third model size pg1 <ref:2606.14302#pg1>. That’s pretty substantial when you factor in the scaling differences between the models they tested.

Jane: It’s not just about success rate; they also look at how reliable these progress estimates are through metrics like Intermediate Discrimination, which measures the gap between successful and failed trajectories.

Lu: Their IntermDisc metric shows that RePro agents have substantially higher discrimination, hitting values like six point three seven for a one-and-a-half billion parameter model and up to eleven point four for a seven billion parameter model pg13 <ref:2606.14302#pg1>. That suggests the agent is really getting better at judging its own performance quality.

Meng: So, it sounds like they’re not just making the agents succeed more often; they’re making them smarter about *why* they are succeeding or failing step-by-step. That moves past just brute force execution, right?

Tom: It does. And we have to talk about what this means for the practical side of things, especially when you think about how these agents will be used in complex, real-world applications that require long sequences of decisions.

Jane: The paper lays out the mechanism clearly: the forward-then-reflect rollout gives them a structured way to build that internal progress signal through this composite reward system.

Title and authors: Lu: They also highlight something important about failure recognition. Even when the overall task fails, their learned progress signal still shows meaningful variation and can reflect setbacks during those failed trajectories pg7. That’s a nice piece of information for building agents that are more robust to errors.

Meng: From an engineering standpoint, having these per-step training signals instead of just one final outcome reward is much better for debugging the learning process itself, because you get feedback at every single stage.

Tom: So, we’ve covered the setup and the core results showing performance gains across different model sizes and tasks like WebShop. But what’s the big picture implication here for autonomous systems?

Jane: It suggests that fostering this internal cognition—this ability to track progress—isn't just useful for executing a single task well; it’s crucial if you want agents to handle genuinely long-horizon problems autonomously.

Lu: The real potential is in moving toward self-improving agents where they can use these retrospective signals to refine their own internal models over time, rather than needing constant human intervention.

Meng: I see the value in that for deployment; if an agent can reliably estimate its progress, we need to trust it more when it's operating in environments where we can’t always watch every single action.

Tom: So, to wrap up on this paper, "Retrospective Progress-Aware Self-Refinement for LLM Agent Training," RePro shows that training agents using a forward-then-reflect approach with retrospective feedback can give them the progress awareness they currently lack.

Jane: It moves the goalposts from just executing actions to building internal self-assessment capabilities through structured, retrospective learning.

Lu: It opens up avenues for truly self-reflective agents that can manage complex sequences without needing continuous external supervision or an extra reward model to guide their step-wise decisions pg9.

Meng: It’s promising because it shows a way to build reliability directly into the agent's decision-making loop through this structured feedback.

Tom: That’s the gist of RePro, showing how learning from completed trajectories can help agents develop better internal progress tracking. We gotta take a quick break before we look at what’s next on the arXiv feed.

The paper's summary: Tom: So, we've looked at the mechanics of how this works, but what does the actual summary tell us about why this is a big deal for AI agents?

Jane: Well, the main point is that these agents are currently really bad at knowing if they’re doing okay on a long task because they don’t have that internal progress check.

Tom: Right, and this paper says we can teach them to do that by using the results from when they actually finish a task as a kind of teacher for their own future steps.

Lu: It's about getting metacognition in there. Instead of relying on just prompting them online, which Tom said actually hurts performance, RePro uses those final outcomes to show the agent what good progress looks like at every single step.

Meng: So it’s like giving them a scorecard after the race so they can adjust their running strategy for the next lap?

Jane: Exactly. It shows that this isn't just some clever trick; it requires dedicated training, using those retrospective trajectories to learn what good progress looks like.

Tom: And the numbers on WebShop were pretty interesting, showing real gains in success rates across different model sizes, which is a big deal for scalability.

Jane: But it’s not just about winning the race; they also showed that these agents get much better at telling the difference between a task that succeeded and one that failed.

Lu: That's because of this metric called Intermediate Discrimination, which measures how much better the successful attempts are compared to the ones that didn't finish well. The scores they got were pretty high for even their biggest models.

Tom: So, if we think about what this means for us in the real world, it suggests that future agents won't just be good at one thing; they’ll be better at managing long-term plans because they can self-correct based on their own internal assessment of where they stand.

Jane: It opens up a path for these agents to handle genuinely complex, multi-stage problems without constantly needing a human watching over every single decision.

Tom: That’s the big picture—moving toward systems that can actually manage long-horizon tasks autonomously by learning to judge their own internal journey.

The paper's improvements: Tom: So, we’ve seen how RePro actually works step by step, but what are the specific improvements they claim this method brings to agents?

Jane: The authors are focusing on how their new training signals make the agents better at judging their own progress, not just getting them to finish the task.

Tom: They’re talking about this composite reward structure that includes "progress shaping" and "onlineretrospective alignment," which gives those per-step signals something to work with.

Lu: It means that instead of just one big score at the end, every single action gets feedback based on how well it aligns with where the agent *should* be according to its later retrospective assessment.

Meng: So if an agent makes a bad move early on, this reward system should immediately tell it something is off without waiting for the final result?

Jane: That’s right. It helps them learn to adjust their strategy in real-time based on what they *think* the end goal requires.

Tom: They also highlight this IntermDisc metric again, showing that agents trained with RePro can really tell the difference between a good path and a bad one much more reliably than before.

Jane: That’s important because it means we have a better way to measure whether an agent is developing actual judgment or just following a fixed script.

Tom: And here’s something I like—they found that even when the whole task fails, the progress signal still shows some variation, meaning the agents learn to recognize when things are going sideways, even if they don't succeed overall.

Lu: That robust recognition of setbacks is key; it means the learning isn't just tied to a perfect outcome.

Meng: From an engineering standpoint, that makes debugging much cleaner because you can pinpoint exactly which step caused the deviation from the expected progress curve.

Jane: It suggests that this approach is more stable for training agents in messy, real-world situations where things don't always go according to plan.

Tom: So, these improvements show that we’re moving beyond just getting a better final score; we’re building agents with a more reliable internal sense of direction and self-awareness.

Lu: This kind of internal cognition is what makes the next generation of AI systems really interesting because they could handle much more unpredictable environments.

Conclusion: Tom: So we’ve covered how RePro actually functions and what those specific training improvements are, but where does this all lead in terms of what we know about AI agents?

Jane: It suggests that building agents capable of long-term planning isn't just about giving them more memory; it's about giving them the internal ability to judge their own journey.

Lu: The authors show that if you train an agent to use its own final results as a progress check, you get a much better sense of what’s successful versus what wasn't.

Meng: From an engineering view, that makes the training process more robust because the feedback loop is self-contained and doesn't rely on some external oracle for every single decision.

Lalam: For me, this capability means we can build AI assistants that are truly self-improving in a cultural sense; they won't just follow instructions, they'll learn to evaluate their own competence.

Tom: Exactly. It changes the goal from simply executing a plan to creating agents that can refine their own strategy based on retrospective knowledge.

Jane: And the core idea behind this whole paper, "Retrospective Progress-Aware Self-Refinement for LLM Agent Training," is making that internal self-assessment possible through structured rollout.

Lu: It’s promising because it shows a way to foster that internal cognition without needing continuous human supervision or another massive reward model just to track progress.

Meng: The limitation they mention, though, is that this setup requires that specific forward-then-reflect training phase to get started, so it's not an immediate plug-and-play solution for every agent.

Tom: Right, so it’s a powerful tool for building highly capable agents on complex tasks like long-horizon problem solving.

Jane: It opens up avenues for truly autonomous systems that can manage intricate sequences without needing constant external guidance to keep them on track.

Lalam: This kind of self-awareness is what will make AI feel more like a partner than just a tool, and that really matters for how we interact with these systems.

Tom: We’ve seen the mechanics and the results from "Retrospective Progress-Aware Self-Refinement for LLM Agent Training," and it’s clear this is moving us toward smarter, more self-aware AI.

Jane: It's a big step in teaching agents how to think about their own progress on long, winding paths.

Lu: We’re excited to see what other papers explore next that might build on this idea of internal reflection.

Xinbei Ma, Congmin Zheng, Jiyang Qiu, Jiale Hong, Yao Yao, Xiangmou Qu, Jiaxin Yin, Xingyu Lou†, Jun Wang†, Weiwen Liu, Weinan Zhang, Zhuosheng Zhang†, Hai Zhao†

Shanghai Jiao Tong University · OPPO Research Institute

cs.CL

Submitted: 2026-06-12

Updated: 2026-10-05

Code: https://github.com/mpSchrader/gym-sokoban

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 89/100

The gist: LLM-based agents trained with reinforcement learning optimize step-wise action prediction but lack metacognitive awareness of task progress, inducing a gap that hinders longhorizon scaling.

Key concepts

Forward-then-Reflect Paradigm
This is the core working pattern where an agent first executes a task while generating an online estimate of its completion percentage. After finishing, it then uses a retrospective prompt to re-assess its progress at every step based on the final result. This cycle teaches the agent to bridge the gap between real-time action and final outcome knowledge.
Progress Reward Shaping
The reward function is modified to include intermediate signals beyond just task success. It incorporates a component that specifically shapes the difference between consecutive progress estimates (rp(t)), encouraging the agent to learn meaningful step-wise improvements rather than just focusing on the final goal.
IntermDisc
This metric measures how well an agent can distinguish between successful and failed trajectories based on their final progress. It calculates the gap between the average final progress achieved by successful runs versus those that failed, serving as a key measure of whether the agent has developed true progress awareness.

Terminology

Summary

LLM-based agents trained with reinforcement learning optimize step-wise action prediction but lack metacognitive awareness of task progress, inducing a gap that hinders longhorizon scaling. The gist: RePro trains agents to self-generate progress signals via a forward-then-reflect rollout paradigm to learn metacognition from retrospective trajectory outcomes.

Pilot Study Findings

The pilot study investigated the potential of progress awareness and revealed a clear asymmetry between online prompting and retrospective demonstrations, showing that progress awareness requires dedicated training.<ref:2606.14302v1[pg3] Online progress prompting hurts performance while retrospective demonstrations help, yet this capability cannot emerge from outcomereward training alone.<ref:2606.14302v1[pg3] The asymmetry reveals that progress awareness is beneficial yet cannot be reliably elicited by prompting alone, motivating a dedicated training approach to learn progress assessment from retrospective trajectory outcomes.<ref:2606.14302v1[pg3]

RePro Framework

RePro is a two-phase framework designed to learn progress awareness from retrospective trajectory outcomes and use it as an auxiliary signal for agent training.<ref:2606.14302v1[pg3] It augments trajectories with a forward-then-reflect paradigm: the agent first forward to execute the task while generating online progress estimates; once completing the task and knowing the outcome, the agent retrospectively reassesses its step-wise progress anchored by the final result.<ref:2606.14302v1[pg4]

The framework consists of two main stages:

  1. Retrospection Warmup: This stage teaches reflection format from minimal external demonstrations, using a small set of external-LLM demonstrations to teach the agent the forward-then-reflect format for retrospective reflection.<ref:2606.14302v1[pg4]

  2. RePro-PO: This stage leverages the retrospective progress signal by a composite reward, comprising retrospective progress shaping, onlineretrospective alignment, and format regularization, producing per-step training signals that complement the sparse outcome reward.<ref:2606.14302v1[pg4]

Forward-then-Reflect Paradigm

The working pattern of the agent is redefined as forward-then-reflect.<ref:2606.14302v1[pg5]

Forward Execution involves the agent generating an online self-assessment and an action conditioned on the interaction history, denoted as (˜pt, at) ∼ πθ(· ht), where p˜t ∈ [0, 100] is the online self-assessment of completion percentage.<ref:2606.14302v1[pg5]

Retrospective Reflection involves prompting the agent with a retrospective reflection prompt to re-assess progress at each step given the complete trajectory and outcome, denoted as (p1,..., pT) ∼ π retroθ(· τ, r), where pt ∈ [0, 100] is the agent’s retrospective self-assessed progress of task completion at step t.<ref:2606.14302v1[pg5] The trajectory is then re-organized by replacing each online estimate p˜t with the corresponding retrospective estimate pt: τretro = (g, [(ot, at, pt)]T) <ref:2606.14302v1[pg5]

Progress Reward Shaping

The reward function integrates the outcome task performance with progress-specific objectives: R(τ) = X T Σ t=1 rt + wb · Rformat.<ref:2606.14302v1[pg9] For each step, progress reflection provides intermediate signals beyond sparse outcome reward: rt = renv(t) + rp(t) + ralign(t) + rformat(t), where rp shapes step-wise progress differences, ralign drives online predictions toward retrospective quality, and rformat is based on format validity.<ref:2606.14302v1[pg10] Specifically, rp(t) = β · (pt+1 − pt), and ralign(t) = 1/T α · max(0, 1 − pt − p˜t).

Empirical Validation

RePro achieves consistent performance gains across three benchmarks and three model scales. On WebShop, RePro improves the absolute task success rate over baseline training by +8.98%, +11.57%, and +5.82% across different model sizes.<ref:2606.14302v1[pg6] RePro generalizes across model scales and task environments, exhibiting consistent gains across both model scales and environments.<ref:2606.14302v1[pg7] Progress quality analysis further shows that RePro agents develop metacognitive awareness: their progress estimates discriminate successful from failed trajectories significantly better than baselines.<ref:2606.14302v1[pg7]

Progress Prediction Metrics

Progress predictions are evaluated through behavioral properties and outcome alignment, specifically measuring Format Rate, Intermediate Discrimination (IntermDisc), Temporal Correlation (Corr), Monotonicity Rate (Mono), Decline Rate (Decline), and Plateau Rate.<ref:2606.14302v1[pg12] The critical metric for outcome-based tasks is IntermDisc, which measures Gap between mean final progress of successful versus failed trajectories: IntermDisc = E[pT rT ≥ 9.0]−E[pT rT < 1.0].<ref:2606.14302v1[pg13] RePro achieves substantially higher discrimination: 6.37 (1.5B), 7 (3B), and 11.4 (7B).

Ultimately, these findings demonstrate the potential of progress awareness not only for execution, but also for fostering internal cognition within long-horizon tasks, opening promising avenues for future research in autonomous selfreflective agents.<ref:2606.14302v1[pg9] The paper concludes by noting that RePro enables agents to incorporate progress-aware feedback without relying on continuous external supervision or an additional reward model.<ref:2606.14302v1[pg9]

REFERENCES

Emre Can Acikgoz, Cheng Qian, Heng Ji, Dilek Hakkani-Tür, and Gokhan Tur. 2025. Selfimproving llm agents at test-time. arXiv preprint arXiv:2510.07841.<ref:2606.14302v1[pg9]

Marc-Antoine Allard, Arnaud Teinturier, Victor Xing, and Gautier Viaud. 2026. Experiential reflective learning for self-improving llm agents. arXiv preprint arXiv:2603.24639.<ref:2606.14302v1[pg11]

Shelly Bensal, Umar Jamil, Christopher Bryant, Melisa Russak, Kiran Kamble, Dmytro Mozolevskyi, Muayad Ali, and Waseem AlShikh. 2025. Reflect, retry, reward: Self-improving llms via reinforcement learning. arXiv preprint arXiv:2505.24726.<ref:2606.14302v1[pg11]

Yuxuan Cai, Yipeng Hao, Jie Zhou, Hang Yan, Zhikai Lei, Rui Zheng, Zhenhua Han, Yutao Yang, Junsong Li, Qianjun Pan, Tianyu Huai et al. 2026. A benchmark for self-evolving agents via experiencedriven lifelong learning. arXiv preprint arXiv:2603.19685.<ref:2606.14302v1[pg11]

Zouying Cao, Runze Wang, Yifei Yang, Xinbei Ma, Xiaoyong Zhu, Bo Zheng, and Hai Zhao. 2025. Progrm: Build better gui agents with progress rewards. arXiv preprint arXiv:2505.18121.<ref:2606.14302v1[pg9]

Qizheng Zhang, Changran Hu, Shubhangi Upasani, Boyuan Ma, Fenglu Hong, Vamsidhar Kamanuru, Jay Rainton et al. 2025b. Agentic context engineering: Evolving contexts for self-improving language models. arXiv preprint arXiv:2510.04618.<ref:2606.14302v1[pg9]

Shiqi Zhang, Xinbei Ma, Yunqing Xu, Zouying Cao, Pengrui Lu et al. 2025c. Paracook: On time-efficient planning for multi-agent systems. Preprint, arXiv:2510.11608.<ref:2606.14302v1[pg9]

Xiuhui Zhang, Ning Gao, Xingyu Jiang, Yihui Chen et al. 2026b. Progress reward model for reinforcement learning via large language models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems.

Improvements for AI systems

  1. Bold header: Retrospective Progress-Aware Training (RePro) Framework

This framework trains agents to self-generate progress signals via a forward-then-reflect rollout paradigm, enabling agents to learn selfgenerated progress metacognition by using retrospective trajectory outcomes as supervision instead of relying on continuous external supervision.

  1. Bold header: Enhanced Long-Horizon Task Execution

The improved AI system can execute long-horizon tasks with significantly higher success rates, achieving gains up to 12% absolute success rate improvements on benchmarks like WebShop, ALFWorld, and Sokoban compared to baselines.

  1. Bold header: Reliable Progress Estimation for Policy Optimization

Agents will generate more reliable progress estimates by utilizing a composite reward that includes progress shaping and onlineretrospective alignment, which produces per-step training signals that complement the sparse outcome reward.

  1. Bold header: Improved Intermediate Discrimination

The agent's ability to distinguish between successful and failed trajectories will be significantly enhanced, evidenced by a higher Intermediate Discrimination (IntermDisc) metric, which measures the gap between the mean final progress of successful versus failed trajectories.

  1. Bold header: Robust Setback Recognition in Failure

The system will learn to recognize setbacks even when the task fails by achieving non-trivial decline rates on failed trajectories, indicating that the learned progress signal retains meaningful variation and can reflect setbacks even when the task fails.

Sources

Related papers