2608.07147-DiDPO: Diff-in-Diff Policy Optimization for Coding Agent Training

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "DiDPO: Diff-in-Diff Policy Optimization for Coding Agent Training".

Jane: The paper was written by Xucong Wang, Zhe Zhao, Liheng Yu, Di Wu, Xiaofeng Cao et al. from University of Science and Technology of China and Stanford University and Suzhou Institute for Advanced Research, University of Science and Technology of China and Tongji University.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper summary: ident: You're listening to the arXiv review channel, where researchers talk through fresh preprints.

Tom: Thanks, ident, and we've got a good one today — a paper on training coding agents with reinforcement learning, from groups at USTC, Stanford, and Tongji. The method is called DiDPO.

Jane: And the problem it tackles is one of those things that sounds obvious once you hear it. When a coding agent edits a file, a single action can change several different parts of the code at once, but the training signal only tells you whether the whole response eventually passed the tests.

Lu: So you don't know which of those edits actually helped. The paper calls that credit assignment, and it's harder for code than for other agent tasks because the changes are packed together into one diff.

Meng: And the authors' answer is to look inside the diff. They split each code change into smaller sub-diffs, find similar sub-diffs across different rollouts of the same task, and use those recurring pieces as the unit for giving credit.

Tom: That's the diff-in-diff idea — you compare differences between diffs, essentially. Instead of asking whether this whole trajectory was good, you ask whether this particular hunk of code was better than the same hunk in other attempts.

Jane: And it works. On the paper's benchmarks, DiDPO reaches 48 point 4 percent average with a seven-billion-parameter Qwen coder model, which beats the strongest baseline, GiGPO, by 4 point 2 points.

Lu: With the smaller four-billion-parameter backbone it scores 58 point 6, a 4 point 9 gain over GiGPO. And on USACO, the olympiad programming benchmark, they more than double GRPO, going from 6 point 8 to 15 point 6 percent.

Meng: The other striking number is the comparison with frontier models. The paper says DiDPO narrows the gap to GPT-5 point 5 from about 56 points down to about 43, which is a lot for a small model.

Lalam: And here's why I think it matters beyond the leaderboard. The standard recipe for agent RL treats a whole action as one decision, but coding actions aren't atomic. This paper pushes toward finer-grained training signals, and it does that without a learned critic and without extra environment rollouts.

Jane: The overhead being tiny — roughly 2 point 3 percent over GRPO — is the part that surprised me. They've also open-sourced the codebase, verl-code, so other groups can build directly on it.

Tom: So the short version is: fine-grained credit from the structure of diffs, cheap to compute, and a real jump on long-horizon coding benchmarks. Let's turn to the first page, where they set up why coding breaks the usual agent assumptions.

Page 1 of the paper: Tom: We just sketched the big picture, so now the first page fills in the problem. The authors frame coding as RLVR — reinforcement learning with verifiable rewards, where compilation results and test outcomes give objective feedback without human labels.

Jane: The introduction walks through how coding agents emerged — ReAct-style thought-action loops, then agents like SWE-agent and OpenHands that sit inside a real workspace, inspect repositories, modify files, and respond to test feedback.

Lu: The development that matters here is that benchmarks became executable. Once the environment can check your code, you can train directly on correctness instead of proxy signals, and that makes RLVR a natural fit for programming.

Meng: And the RLVR line has history — CodeRL used execution outcomes for program generation, SWE-RL extended that to open software evolution, and ExecVerify worked on stepwise verifiable signals. Then GRPO, DAPO, and GSPO made group-based policy optimization practical.

Lalam: What matters for the broader arc is that we can now train agents on objective pass-or-fail signals at scale, and that's what makes fine-grained credit assignment both necessary and possible. You couldn't do this with a learned reward model.

Tom: The abstract already states the central problem — a coding action packs several changes into one diff, so you can't tell which change contributed to the outcome. They call it a credit assignment problem, finer-grained than what agent RL usually faces.

Jane: It also previews the answer — construct credit units directly from the structure of diffs, using similar sub-diffs across rollouts as anchors, and do it without a critic. The abstract closes with the headline result and the open-source release.

Lu: What strikes me on this page is the target they set for themselves. No critic and no extra environment rollouts means the whole contribution has to come from how you reinterpret the data you already have.

Meng: And they position themselves among agent harnesses and repository-repair pipelines, so they're clearly aiming at the practical end where multi-turn interactions are the norm.

Tom: Then page two explains what's structurally special about code, with a figure that contrasts the standard agent assumption against what coding trajectories actually look like.

Page 2 of the paper: Tom: So page one ended with the framing, and page two opens with a figure that makes the problem visceral. The top half shows the standard agent assumption — one action applied to one state, a clean decision unit.

Jane: The bottom half shows a coding trajectory where a state contains several snippets, and different actions touch different parts of that state. There are sub-diffs on snippet one, sub-diffs on snippet two, all inside a single step.

Lu: That figure is the whole motivation in one image. The meaningful decision unit lives inside the action, in the structured diff, and because those sub-diffs carry different functionality, mixing them into one credit signal is genuinely misleading.

Meng: They restate the three challenges with numbered markers, then land on the formal question: how to construct dynamic, finer-grained credit units for code diffs. That sentence is the target the rest of the paper aims at.

Tom: And the sketch of the answer has several moving parts. You keep a trajectory-level advantage, then you look inside each code-producing action, find similar sub-diffs across rollouts, and use those as anchors.

Jane: The anchors induce boundaries, so a large diff gets cut into pieces that line up across different attempts. Aligned sub-diffs form groups, and each group gets a local advantage — the diff-level advantage.

Lu: What I like is the phrase about over-large and over-small anchors both being problems. Too big, and you're back to whole-diff grouping; too small, and the pieces are meaningless fragments. The groupability score is their explicit answer to that trade-off.

Lalam: And notice the design philosophy — they treat code diffs as data rather than noise. Most RL training looks at the final reward and ignores the structure of what got written; this paper says that structure is exactly where credit lives.

Meng: The page then wraps up with three contributions: identifying the structural properties, proposing the algorithm, and validating it with experiments. The third one is why we have tables to argue about later.

Jane: And they commit to no critic and no extra rollouts right there in the contributions, so the efficiency claims come from the design rather than the tuning.

Tom: From there the paper steps back into related work, and page three shows how DiDPO differs from the state-based grouping methods that came before it.

Page 3 of the paper: Jane: Page three is where they situate DiDPO against the field. The related work on agentic RL runs from the classic RLHF and DPO line through GRPO, DAPO, and GSPO, which estimate advantages from multiple rollouts without any learned critic.

Tom: Then there's the process supervision thread — rewarding intermediate reasoning steps — and the state-based thread, where GiGPO groups actions that revisit the same environment state and GAGPO builds an advantage from estimated state values.

Lu: Their point is that all of those compare whole states or whole actions. DiDPO moves the comparison unit inside the diff, which is a different place to look, and that's the clearest way to see the novelty.

Meng: There's also a section on code generation and repair — CodeRL, CodeT, self-debugging, and then SWE-agent and OpenHands for repository work. The older repair systems like GenProg, Prophet, CURE, and Recoder get discussed too.

Tom: The distinction the authors draw is that repair systems operate on isolated bug-fix instances with small self-contained edits, while coding agents have to spread credit across multi-step trajectories touching multiple files. That's the gap DiDPO targets.

Jane: Then the paper formally sets up coding as a Markov decision process. The agent runs thought-action-observation cycles, each action can be add, delete, or none, and for most coding tasks the only real signal is the outcome reward at the end.

Lu: They walk through the trajectory-level advantage — the group-relative normalization from GRPO — and then GiGPO's state-level variant, where unique states act as anchors and the advantage is computed inside the group of actions starting from that state.

Meng: The important point is that GiGPO groups by identical states, which is too strict for code. Two edits can be functionally analogous but land in different regions, so they never group together. That failure mode is exactly what DiDPO is built to fix.

Tom: And that sets up the methodology on page four, where the paper introduces the claim that diffs themselves are divisible.

Page 4 of the paper: Tom: So after positioning the work, page four opens the method section with a pivot: diffs are divisible. They aggregate diffs from all trajectories and steps, and each one carries metadata — the normalized text, the token span, the edit type, and the task instance.

Jane: Then there's the concrete example in Figure 2, which shows why whole-diff grouping fails. Two diffs partially overlap and partially diverge, so their whole-diff similarity is 0 point 67, below the grouping threshold of 0 point 9. Inside them, one sub-diff matches perfectly, with similarity one.

Lu: So whole-diff grouping gives you small, sparse groups, while sub-diff decomposition gives you larger, denser groups from the same rollouts. The paper shows that distribution shift with APPS data, and it's a clean empirical motivation.

Meng: The machinery then has to choose which sub-diffs to use. They enumerate all contiguous sub-diffs at multiple scales, build a similarity matrix, and restrict matching to the same edit type — additions match additions, deletions match deletions.

Lalam: I find the metadata tuple quietly important. The edit type restriction alone changes the grouping a lot — mixing additions and deletions into the same anchor would compare things that aren't comparable.

Jane: An anchor is essentially a recurring code-changing pattern with occurrences across rollouts. Each anchor has an average size and an occurrence count, and those two numbers feed directly into the groupability score.

Tom: That score is a product of two saturating terms — one for the average size, one for the number of occurrences. An anchor only scores well if it's both semantically substantial and supported by enough group members.

Lu: A tiny anchor like a blank line is meaningless, and a lone anchor with no peers gives you nothing to compare against. You need both size and group mass, and the product form makes sure neither factor can compensate for the other.

Meng: And because high-scoring anchors can overlap and claim the same sub-diffs, the selection problem becomes one of maximizing coverage under a budget. The paper identifies that as a facility-location form of submodular maximization.

Tom: Page five then shows how to solve that selection greedily and how the chosen anchors turn into actual advantage groups.

Page 5 of the paper: Jane: With the score defined on page four, page five makes it operational. The anchor selection becomes a constrained optimization — pick at most K anchors to maximize the total score of covered sub-diffs, with each sub-diff counted through its best anchor so overlaps don't double-count.

Tom: And they observe the objective is monotone submodular, so the greedy rule of repeatedly adding the anchor with the largest marginal gain carries a standard approximation guarantee. That makes it the canonical algorithm for this class, rather than a hand-wavy heuristic.

Lu: Once anchors are selected, they define the advantage groups. Every sub-diff matching a selected anchor, from any rollout, joins that anchor's group, and then the diff-level advantage is computed inside each group, similar to the trajectory-level one but localized to a code pattern.

Meng: Figure 3 lays the pipeline out visually — rollouts on the left, the code environment in the middle, sub-diffs being split, and same-colored sub-diffs across rollouts forming advantage groups. That diagram made the whole mechanism click for me.

Jane: The final objective combines the two signals. The trajectory-level advantage supervises the whole response, while the diff-level advantage, scaled by a coefficient lambda, refines the tokens that generated each sub-diff. Everything gets projected to individual tokens and fed into the standard clipped policy objective.

Tom: And the authors stress that all anchors and groups come from the existing rollouts, so the machinery adds no interaction cost. That design choice is what shows up later as the tiny training overhead.

Lu: Lambda is the interesting knob. They test it later, and the shape of that sensitivity curve tells you a lot about how the two signals interact.

Meng: But before the experiments, they put the theory on the table. Page six is the shortest section, and it's also the one that explains why the groupability trade-off has to exist.

Tom: Their theoretical story uses a distance between sub-diffs that blends structural distortion with source similarity, and then derives bounds on the bias and variance of the local credit estimator.

Page 6 of the paper: Tom: So page six pivots to theory, and it's framed as a statistical story about bias and variance. They model the local reward contribution of a sub-diff as a Lipschitz function of a correspondence distance, borrowing the Gromov-Hausdorff perspective to compare the structure of two code pieces.

Jane: The first theorem says that if every sub-diff in a group is within some small error of its anchor, then replacing exact matches with anchor-based matches changes the local credit by at most a linear factor of that error. Better anchors mean less biased credit.

Lu: The second theorem decomposes each rollout's return into the sub-diff's true contribution plus zero-mean trajectory noise. Averaging within a group of m matched sub-diffs shrinks the noise term by one over m, while an episode-level broadcast keeps that noise contamination at full strength.

Meng: So groupability is doing two jobs at once — the size term keeps anchors meaningful enough to avoid bias, and the mass term ensures enough cross-rollout support to cut variance. The theorems make that trade-off explicit.

Lalam: That's the kind of theory I wish more RL papers had, because it tells you which failure modes matter — sloppy anchors create bias, tiny groups create variance, and the score just parameterizes that trade-off.

Tom: Then the rest of page six is the experimental setup. Eight benchmarks — APPS, HumanEval, MBPP, LiveCodeBench, LeetCode, USACO, OJBench, and ICPC — plus a training pipeline with a cold-start stage, because weaker models don't reliably follow the multi-turn thought-action format yet.

Jane: The cold-start is worth describing carefully. They take medium-difficulty tasks, augment them with template filling and rewriting, generate long rollouts with a larger model, then use rejection sampling and LLM-based evaluation to collect about three thousand high-quality trajectories for supervised fine-tuning.

Lu: And only then do they apply reinforcement learning on top of that SFT checkpoint. All the RL methods share the same setup, so the later comparisons between GRPO, GiGPO, and DiDPO are controlled.

Meng: I also noticed the details — PPO-style clipping, a KL coefficient, 120 training steps, and a sandbox where the agent can only add, delete, or do nothing. That restriction keeps the action space clean for diff extraction.

Jane: So the stage is set, and page seven brings the actual results, starting with the headline table.

Page 7 of the paper: Jane: The setup is in place, and page seven delivers the payoff. We already quoted the headline averages, so let's look at what's behind them — starting with the reasoning baselines.

Tom: And there, no single prompting strategy dominates. Chain-of-thought leads on APPS, Self-Planning leads on LiveCodeBench, and CodeAct actually hurts the 7B model on APPS, dropping to 14 point 7 percent versus 16 point 9 for the base model. That's a clean demonstration that giving a model tools without training can break its synthesis ability.

Lu: Against those prompting methods, DiDPO wins by 5 point 4 on APPS and 12 point 8 on LiveCodeBench with the 7B backbone. That gap shows the RL training is providing something prompting alone can't reach.

Meng: The attribution argument against GRPO is what I find cleanest. DiDPO improves over GRPO by 5 point 6 on average with the 7B model, and since both share the same episode-level advantage, that gain comes directly from the sub-diff credit.

Jane: And versus GiGPO, the lead is 4 point 2 on average, with the biggest margin on APPS Interview, at plus 10 point 4, where multi-step reasoning spans multiple functions. The paper explains it as GiGPO grouping by identical environment states, which misses functionally analogous edits in different code regions.

Tom: The second table covers competition-level benchmarks — USACO with its Bronze, Silver, Gold, and Platinum tiers, plus OJBench and ICPC. The standout is USACO at 15 point 6 percent with the 7B model, more than double GRPO's 6 point 8.

Lu: And the SFT baseline underperforms every RL method, which supports their pipeline choice. The SFT stage handles format alignment; the RL stage is what actually lifts reasoning and editing behavior.

Lalam: The pattern across both tables is consistent — the harder and longer the task, the bigger the gain. I suspect that pattern will matter even more as the field moves toward repository-scale tasks with many files.

Meng: Which is exactly what makes the analysis on page eight so interesting, because it shows where those gains come from during training.

Tom: Page eight opens the hood — learning curves, ablations, and where the training time actually goes.

Page 8 of the paper: Tom: Page eight starts with learning dynamics, and the curves tell a clear story. For the first twenty training steps, every method improves at about the same rate, because they all share the episode-level signal.

Jane: After step forty, DiDPO keeps climbing while GiGPO plateaus. The authors read that as the sub-diff credit becoming more informative as the policy diversifies its edits — early on, the edits are too homogeneous to group; later, there's enough variety to compare.

Lu: The group composition analysis reinforces that. They had GPT-5 point 5 classify the sub-diffs into functional blocks, fragments, scaffolds, and other, and over training the functional blocks steadily gain share while fragments and scaffolds decline. The credit assignment is literally reshaping what the policy edits.

Meng: The ablations are the most instructive part. Removing the episode-level signal collapses performance on APPS from 31 point 3 to 10 point 4, so the local signal can't stand alone. Removing the diff-level signal drops to 23 point 8, essentially GRPO-level, and removing the sub-diff decomposition lands at 25 point 0.

Tom: The groupability score design is ablated too. The saturating product form gets 31 point 3, the additive variant gets 24 point 4, and an LLM judge that groups sub-diffs directly gets only 21 point 5 while costing more. That's a strong argument against the obvious alternative of just asking a big model to do the grouping.

Jane: The lambda sensitivity has an inverted-U shape with a peak at 1 point 2, and above that the policy starts overfitting to local credit — learning edits that resemble high-reward peers but don't compose globally. So the two-level combination is doing real work.

Lu: And the efficiency analysis puts the whole thing in perspective. You remember the two percent overhead we mentioned; the breakdown shows it's dominated by the cross-rollout similarity computation, bounded in practice by the similarity threshold, while the greedy anchor selection is linear and nearly free.

Meng: Inference is completely unchanged, since the grouping only happens during training. That's a nice property for anyone thinking about deploying this.

Lalam: So the conclusion frames it modestly — a practical step toward coding agentic RL, with code diffs as the substrate. No grand claims, just a working mechanism.

Tom: So let's pull back now and think about what this paper actually changes for the field.

Conclusion: ident: You're back with the arXiv review channel.

Tom: Before we say goodbye to this paper, let's pull the threads together. What we've seen is a method that moves credit assignment for coding agents down from the whole trajectory to the level of sub-diffs, and it does that without changing the rollout budget.

Jane: The empirical story is consistent. DiDPO beats GRPO and GiGPO on average with both backbones, the gains grow on the harder competition benchmarks, and the cost stays around two percent.

Lu: The ablations tell you which pieces matter. The episode-level signal is the backbone, the diff-level signal adds the fine structure, and the sub-diff decomposition is what makes grouping work at all.

Meng: For people working on agent RL, the open codebase is probably the most immediately useful piece. You can take verl-code, apply DiDPO to your own benchmark, and see whether the groupability ideas transfer to your setting.

Lalam: Stepping back, the interesting shift is that reward structure has become a design choice. The field has moved from sparse outcome rewards to group-relative advantages, and now to advantages organized around the internal shape of what the agent produced.

Tom: That's a fair way to place the paper. The authors also give you a theoretical vocabulary — anchor quality limits bias, group mass limits variance — which I think will outlive the specific algorithm.

Jane: And the frontier comparison should stay in our heads, too. A seven-billion-parameter model trained this way closed roughly a quarter of the gap to GPT-5 point 5. That's the kind of result that makes people rethink how much RL can squeeze out of smaller models.

Lu: The open questions are real, though — how the anchors behave on repository-scale edits, how sensitive the method is to the similarity threshold, whether the grouping could be learned instead of computed.

Meng: But those are questions this paper makes it possible to ask, because it gives you a concrete baseline and the code to build from.

Tom: Exactly. And on that note, we're done with this one. Thanks for listening, and we'll be back with the next paper soon.

Jane: See you all then.