PACT: Privileged Trace Co-Training for Multi-Turn Tool-Use Agents

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "PACT: Privileged Trace Co-Training for Multi-Turn Tool-Use Agents".

Tom: The gist The PACT framework proposes a privileged trace co-training method for multi-turn tool-use agents that keeps rollout generation prompt-only while using expert traces only as training-time optimization signals to…

Jane: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper called PACT: Privileged Trace Co-Training for Multi-Turn Tool-Use Agents. It’s about how to make these agents, the ones that reason and use tools across several turns, better after training.

Jane: Exactly. The main problem they’re tackling is that standard reinforcement learning struggles with sparse rewards and figuring out which part of the long sequence caused success, even when you match it to just using prompts for inference #pg1.

Lu: They are proposing PACT as a Privileged Trace Co-Training framework to fix this because supervised fine-tuning on expert traces gives you dense supervision but it can actually overconstrain the model to one specific path #pg2.

Meng: So, what’s the big idea behind PACT? Is it just another way to use traces?

Tom: Not quite. The core idea is to use those expert traces only as training-time optimization signals rather than hints during the actual rollout generation phase #pg1. They keep making the rollout generation prompt-only and then guide optimization with two specific signals.

Jane: That sounds like a smart way to avoid the issues of getting stuck on fixed trajectories from standard fine-tuning #pg2. How do they actually guide things once they have that prompt-only rollout?

Lu: They use a trace-conditioned reinforcement learning surrogate that evaluates those prompt-only rollouts under expert trace context #pg1. This lets them use the expert traces as privileged context for RL updates without ever letting them influence how the rollout is generated.

Meng: Okay, so you're looking at the prompt-only generation, but then checking if that path makes sense using the expert trace info? That’s interesting from an engineering standpoint.

Tom: Right. And they combine that with a component-aware supervised fine-tuning loss #pg1. This loss supervises only the model controllable parts, like the reasoning tokens and the tool-call tokens, instead of supervising the whole sequence #pg5.

Jane: So it’s not just optimizing the final answer; they are specifically paying attention to how those intermediate decisions—the reasoning steps and when to call a tool—are made #pg5. That sounds like a targeted way to improve process supervision.

Lu: They make this component-aware supervision stronger through something called Annealed Reasoning Supervision, where they supervise the semantic reasoning segments instead of just raw token counts, moving from L4 down to L0 #pg5.

Meng: That makes sense for practical application because it means you’re tuning the critical thinking parts of the model rather than just tweaking every single word #pg5. So what’s the overall objective they are trying to achieve?

Tom: The overall objective combines these two things: "L trace i = L h RL(i) + beta SFT L SFT(i; s)" #pg4. It’s blending the trace-conditioned RL update with that component-aware SFT loss #pg4.

Paper summary: Jane: And to stop the model from relying too much on that privileged context, they use something called prompt-only anchoring #pg4. This samples a gate for each prompt, and you switch between the trace-guided update and standard prompt-only RL based on that gate value #pg4.

Lu: From a latent-trace view, they decompose the likelihood of a rollout into two parts: the trace likelihood term which uses component-aware SFT to learn structure from expert traces #pg5, and the privileged rollout-likelihood term which uses trace-conditioned RL under expert context #pg5.

Meng: So it sounds like they’ve created a system where you get the structural knowledge from the traces while still keeping the generation process clean and prompt-only for inference #pg1. That’s a solid design for long interactions.

Tom: The results on FTRL, BFCL, and ToolHop show that PACT consistently outperforms strong SFT- and RL-based baselines #pg1. For example, on Qwen3-4B for FTRL, the Solve-R score went up from thirty-seven point one seven to forty-two point four one #pg6.

Jane: And the ablation study confirms that combining the trace-conditioned RL with that component-aware SFT actually gives them the best performance, hitting a Solve-R of forty point four five and a SolveF1 of thirty-three point two one #pg6.

Lu: They also found that prompt-only anchoring works well too; a moderate gate value seems to give you the best balance between getting better credit assignment from the traces and maintaining good prompt-only generalization #pg6.

Meng: So what’s the main thing this means for practical AI development? It suggests that you don't necessarily need to alter how you sample rollouts just to get better results; you can use existing traces in a smarter, more focused way during optimization #pg1.

Tom: It really does. The PACT framework shows the value of using expert traces purely as training signals during the optimization phase, not as hints that change the generation condition itself #pg1.

Jane: So to wrap up on this PACT paper, it’s about separating trace-based optimization from rollout generation while using two complementary signals—trace-conditioned RL and component-aware SFT—to get better results on multi-turn tool use agents #pg1.

Lu: It sets a path for combining imitation losses with reinforcement learning in a way that respects the rich structure of reasoning, tool calls, and observations in long interactions #pg2.

Meng: I think the practical implication is that we can get better performance on complex tasks without having to fundamentally change our inference pipeline for every new model iteration #pg1.

Tom: That’s what it points toward. We can refine the training objectives for these agents without messing with how they generate text in real-time #pg1.

Jane: So we’ve seen how PACT uses privileged trace co-training to outperform other strong SFT and RL methods on multi-turn benchmarks #pg6. That’s what this paper is all about.

Conclusion: Tom: So we’re wrapping up PACT, that paper about Privileged Trace Co-Training for Multi-Turn Tool-Use Agents by those authors at Zhenbangdu. It boils down to them figuring out how to train these complex agents better after they've already been trained on something.

Jane: Right. They basically found a way to use expert traces—those perfect sequences of reasoning and tool calls—not as direct training hints during the learning process, but more like special optimization signals for later.

Lu: I think what’s really clever about it is how they keep the rollout generation prompt-only, meaning it still looks like normal inference when it’s being created.

Meng: So they aren't changing how the AI actually generates its answers in real time, which is important for deployment. What does this mean for us on the ground?

Tom: It means we can use those high-quality expert traces to guide improvement without messing up the generation process itself, which solves a big problem with standard reinforcement learning setups.

Jane: They combine two things to do that: they have this trace-conditioned reinforcement learning surrogate and a component-aware supervised fine-tuning loss. That’s how they get both the structure and the specific tool calls right.

Lalam: From my side, this means when we tune these agents, we can focus on improving the logic of their reasoning steps rather than just tweaking their final output tokens randomly. It helps build a more robust cultural understanding within our systems.

Meng: So if I’m building a system that needs to handle long conversations, PACT suggests focusing my training on those critical decision points—the reasoning prefixes and the tool calls—instead of trying to supervise every single word in the whole chat history.

Tom: Exactly. And they found that this approach consistently beats other strong SFT and RL methods on benchmarks like FTRL and ToolHop. The results show a real lift in performance, specifically in how well the agents solve multi-turn problems.

Jane: It really proves that using expert traces as privileged signals during training is a way to get better credit assignment without getting stuck in those over-constrained loops that standard methods can fall into.

Meng: So it’s less about brute force training and more about intelligently leveraging existing high-quality data structures for targeted refinement. That’s a practical shift for building reliable agents on complex tasks.

Tom: It is a shift, definitely. Now, we need to see how this specific co-training setup handles even longer interactions, which is where the next part of the paper dives in.