DiDPO: Diff-in-Diff Policy Optimization for Coding Agent Training

arXiv:2608.07147 · cs.AI · Submitted 2026-08-07 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "DiDPO: Diff-in-Diff Policy Optimization for Coding Agent Training".

Jane: The paper was written by Xucong Wang, Zhe Zhao, Liheng Yu, Di Wu, Xiaofeng Cao et al. from University of Science and Technology of China and Stanford University and Suzhou Institute for Advanced Research, University of Science and Technology of China and Tongji University.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper summary: ident: You're listening to the arXiv review channel, where researchers talk through fresh preprints.

Tom: Thanks, ident, and we've got a good one today — a paper on training coding agents with reinforcement learning, from groups at USTC, Stanford, and Tongji. The method is called DiDPO.

Jane: And the problem it tackles is one of those things that sounds obvious once you hear it. When a coding agent edits a file, a single action can change several different parts of the code at once, but the training signal only tells you whether the whole response eventually passed the tests.

Lu: So you don't know which of those edits actually helped. The paper calls that credit assignment, and it's harder for code than for other agent tasks because the changes are packed together into one diff.

Meng: And the authors' answer is to look inside the diff. They split each code change into smaller sub-diffs, find similar sub-diffs across different rollouts of the same task, and use those recurring pieces as the unit for giving credit.

Tom: That's the diff-in-diff idea — you compare differences between diffs, essentially. Instead of asking whether this whole trajectory was good, you ask whether this particular hunk of code was better than the same hunk in other attempts.

Jane: And it works. On the paper's benchmarks, DiDPO reaches 48 point 4 percent average with a seven-billion-parameter Qwen coder model, which beats the strongest baseline, GiGPO, by 4 point 2 points.

Lu: With the smaller four-billion-parameter backbone it scores 58 point 6, a 4 point 9 gain over GiGPO. And on USACO, the olympiad programming benchmark, they more than double GRPO, going from 6 point 8 to 15 point 6 percent.

Meng: The other striking number is the comparison with frontier models. The paper says DiDPO narrows the gap to GPT-5 point 5 from about 56 points down to about 43, which is a lot for a small model.

Lalam: And here's why I think it matters beyond the leaderboard. The standard recipe for agent RL treats a whole action as one decision, but coding actions aren't atomic. This paper pushes toward finer-grained training signals, and it does that without a learned critic and without extra environment rollouts.

Jane: The overhead being tiny — roughly 2 point 3 percent over GRPO — is the part that surprised me. They've also open-sourced the codebase, verl-code, so other groups can build directly on it.

Tom: So the short version is: fine-grained credit from the structure of diffs, cheap to compute, and a real jump on long-horizon coding benchmarks. Let's turn to the first page, where they set up why coding breaks the usual agent assumptions.

Page 1 of the paper: Tom: We just sketched the big picture, so now the first page fills in the problem. The authors frame coding as RLVR — reinforcement learning with verifiable rewards, where compilation results and test outcomes give objective feedback without human labels.

Jane: The introduction walks through how coding agents emerged — ReAct-style thought-action loops, then agents like SWE-agent and OpenHands that sit inside a real workspace, inspect repositories, modify files, and respond to test feedback.

Lu: The development that matters here is that benchmarks became executable. Once the environment can check your code, you can train directly on correctness instead of proxy signals, and that makes RLVR a natural fit for programming.

Meng: And the RLVR line has history — CodeRL used execution outcomes for program generation, SWE-RL extended that to open software evolution, and ExecVerify worked on stepwise verifiable signals. Then GRPO, DAPO, and GSPO made group-based policy optimization practical.

Lalam: What matters for the broader arc is that we can now train agents on objective pass-or-fail signals at scale, and that's what makes fine-grained credit assignment both necessary and possible. You couldn't do this with a learned reward model.

Tom: The abstract already states the central problem — a coding action packs several changes into one diff, so you can't tell which change contributed to the outcome. They call it a credit assignment problem, finer-grained than what agent RL usually faces.

Jane: It also previews the answer — construct credit units directly from the structure of diffs, using similar sub-diffs across rollouts as anchors, and do it without a critic. The abstract closes with the headline result and the open-source release.

Lu: What strikes me on this page is the target they set for themselves. No critic and no extra environment rollouts means the whole contribution has to come from how you reinterpret the data you already have.

Meng: And they position themselves among agent harnesses and repository-repair pipelines, so they're clearly aiming at the practical end where multi-turn interactions are the norm.

Tom: Then page two explains what's structurally special about code, with a figure that contrasts the standard agent assumption against what coding trajectories actually look like.

Page 2 of the paper: Tom: So page one ended with the framing, and page two opens with a figure that makes the problem visceral. The top half shows the standard agent assumption — one action applied to one state, a clean decision unit.

Jane: The bottom half shows a coding trajectory where a state contains several snippets, and different actions touch different parts of that state. There are sub-diffs on snippet one, sub-diffs on snippet two, all inside a single step.

Lu: That figure is the whole motivation in one image. The meaningful decision unit lives inside the action, in the structured diff, and because those sub-diffs carry different functionality, mixing them into one credit signal is genuinely misleading.

Meng: They restate the three challenges with numbered markers, then land on the formal question: how to construct dynamic, finer-grained credit units for code diffs. That sentence is the target the rest of the paper aims at.

Tom: And the sketch of the answer has several moving parts. You keep a trajectory-level advantage, then you look inside each code-producing action, find similar sub-diffs across rollouts, and use those as anchors.

Jane: The anchors induce boundaries, so a large diff gets cut into pieces that line up across different attempts. Aligned sub-diffs form groups, and each group gets a local advantage — the diff-level advantage.

Lu: What I like is the phrase about over-large and over-small anchors both being problems. Too big, and you're back to whole-diff grouping; too small, and the pieces are meaningless fragments. The groupability score is their explicit answer to that trade-off.

Lalam: And notice the design philosophy — they treat code diffs as data rather than noise. Most RL training looks at the final reward and ignores the structure of what got written; this paper says that structure is exactly where credit lives.

Meng: The page then wraps up with three contributions: identifying the structural properties, proposing the algorithm, and validating it with experiments. The third one is why we have tables to argue about later.

Jane: And they commit to no critic and no extra rollouts right there in the contributions, so the efficiency claims come from the design rather than the tuning.

Tom: From there the paper steps back into related work, and page three shows how DiDPO differs from the state-based grouping methods that came before it.

Page 3 of the paper: Jane: Page three is where they situate DiDPO against the field. The related work on agentic RL runs from the classic RLHF and DPO line through GRPO, DAPO, and GSPO, which estimate advantages from multiple rollouts without any learned critic.

Tom: Then there's the process supervision thread — rewarding intermediate reasoning steps — and the state-based thread, where GiGPO groups actions that revisit the same environment state and GAGPO builds an advantage from estimated state values.

Lu: Their point is that all of those compare whole states or whole actions. DiDPO moves the comparison unit inside the diff, which is a different place to look, and that's the clearest way to see the novelty.

Meng: There's also a section on code generation and repair — CodeRL, CodeT, self-debugging, and then SWE-agent and OpenHands for repository work. The older repair systems like GenProg, Prophet, CURE, and Recoder get discussed too.

Tom: The distinction the authors draw is that repair systems operate on isolated bug-fix instances with small self-contained edits, while coding agents have to spread credit across multi-step trajectories touching multiple files. That's the gap DiDPO targets.

Jane: Then the paper formally sets up coding as a Markov decision process. The agent runs thought-action-observation cycles, each action can be add, delete, or none, and for most coding tasks the only real signal is the outcome reward at the end.

Lu: They walk through the trajectory-level advantage — the group-relative normalization from GRPO — and then GiGPO's state-level variant, where unique states act as anchors and the advantage is computed inside the group of actions starting from that state.

Meng: The important point is that GiGPO groups by identical states, which is too strict for code. Two edits can be functionally analogous but land in different regions, so they never group together. That failure mode is exactly what DiDPO is built to fix.

Tom: And that sets up the methodology on page four, where the paper introduces the claim that diffs themselves are divisible.

Page 4 of the paper: Tom: So after positioning the work, page four opens the method section with a pivot: diffs are divisible. They aggregate diffs from all trajectories and steps, and each one carries metadata — the normalized text, the token span, the edit type, and the task instance.

Jane: Then there's the concrete example in Figure 2, which shows why whole-diff grouping fails. Two diffs partially overlap and partially diverge, so their whole-diff similarity is 0 point 67, below the grouping threshold of 0 point 9. Inside them, one sub-diff matches perfectly, with similarity one.

Lu: So whole-diff grouping gives you small, sparse groups, while sub-diff decomposition gives you larger, denser groups from the same rollouts. The paper shows that distribution shift with APPS data, and it's a clean empirical motivation.

Meng: The machinery then has to choose which sub-diffs to use. They enumerate all contiguous sub-diffs at multiple scales, build a similarity matrix, and restrict matching to the same edit type — additions match additions, deletions match deletions.

Lalam: I find the metadata tuple quietly important. The edit type restriction alone changes the grouping a lot — mixing additions and deletions into the same anchor would compare things that aren't comparable.

Jane: An anchor is essentially a recurring code-changing pattern with occurrences across rollouts. Each anchor has an average size and an occurrence count, and those two numbers feed directly into the groupability score.

Tom: That score is a product of two saturating terms — one for the average size, one for the number of occurrences. An anchor only scores well if it's both semantically substantial and supported by enough group members.

Lu: A tiny anchor like a blank line is meaningless, and a lone anchor with no peers gives you nothing to compare against. You need both size and group mass, and the product form makes sure neither factor can compensate for the other.

Meng: And because high-scoring anchors can overlap and claim the same sub-diffs, the selection problem becomes one of maximizing coverage under a budget. The paper identifies that as a facility-location form of submodular maximization.

Tom: Page five then shows how to solve that selection greedily and how the chosen anchors turn into actual advantage groups.

Page 5 of the paper: Jane: With the score defined on page four, page five makes it operational. The anchor selection becomes a constrained optimization — pick at most K anchors to maximize the total score of covered sub-diffs, with each sub-diff counted through its best anchor so overlaps don't double-count.

Tom: And they observe the objective is monotone submodular, so the greedy rule of repeatedly adding the anchor with the largest marginal gain carries a standard approximation guarantee. That makes it the canonical algorithm for this class, rather than a hand-wavy heuristic.

Lu: Once anchors are selected, they define the advantage groups. Every sub-diff matching a selected anchor, from any rollout, joins that anchor's group, and then the diff-level advantage is computed inside each group, similar to the trajectory-level one but localized to a code pattern.

Meng: Figure 3 lays the pipeline out visually — rollouts on the left, the code environment in the middle, sub-diffs being split, and same-colored sub-diffs across rollouts forming advantage groups. That diagram made the whole mechanism click for me.

Jane: The final objective combines the two signals. The trajectory-level advantage supervises the whole response, while the diff-level advantage, scaled by a coefficient lambda, refines the tokens that generated each sub-diff. Everything gets projected to individual tokens and fed into the standard clipped policy objective.

Tom: And the authors stress that all anchors and groups come from the existing rollouts, so the machinery adds no interaction cost. That design choice is what shows up later as the tiny training overhead.

Lu: Lambda is the interesting knob. They test it later, and the shape of that sensitivity curve tells you a lot about how the two signals interact.

Meng: But before the experiments, they put the theory on the table. Page six is the shortest section, and it's also the one that explains why the groupability trade-off has to exist.

Tom: Their theoretical story uses a distance between sub-diffs that blends structural distortion with source similarity, and then derives bounds on the bias and variance of the local credit estimator.

Page 6 of the paper: Tom: So page six pivots to theory, and it's framed as a statistical story about bias and variance. They model the local reward contribution of a sub-diff as a Lipschitz function of a correspondence distance, borrowing the Gromov-Hausdorff perspective to compare the structure of two code pieces.

Jane: The first theorem says that if every sub-diff in a group is within some small error of its anchor, then replacing exact matches with anchor-based matches changes the local credit by at most a linear factor of that error. Better anchors mean less biased credit.

Lu: The second theorem decomposes each rollout's return into the sub-diff's true contribution plus zero-mean trajectory noise. Averaging within a group of m matched sub-diffs shrinks the noise term by one over m, while an episode-level broadcast keeps that noise contamination at full strength.

Meng: So groupability is doing two jobs at once — the size term keeps anchors meaningful enough to avoid bias, and the mass term ensures enough cross-rollout support to cut variance. The theorems make that trade-off explicit.

Lalam: That's the kind of theory I wish more RL papers had, because it tells you which failure modes matter — sloppy anchors create bias, tiny groups create variance, and the score just parameterizes that trade-off.

Tom: Then the rest of page six is the experimental setup. Eight benchmarks — APPS, HumanEval, MBPP, LiveCodeBench, LeetCode, USACO, OJBench, and ICPC — plus a training pipeline with a cold-start stage, because weaker models don't reliably follow the multi-turn thought-action format yet.

Jane: The cold-start is worth describing carefully. They take medium-difficulty tasks, augment them with template filling and rewriting, generate long rollouts with a larger model, then use rejection sampling and LLM-based evaluation to collect about three thousand high-quality trajectories for supervised fine-tuning.

Lu: And only then do they apply reinforcement learning on top of that SFT checkpoint. All the RL methods share the same setup, so the later comparisons between GRPO, GiGPO, and DiDPO are controlled.

Meng: I also noticed the details — PPO-style clipping, a KL coefficient, 120 training steps, and a sandbox where the agent can only add, delete, or do nothing. That restriction keeps the action space clean for diff extraction.

Jane: So the stage is set, and page seven brings the actual results, starting with the headline table.

Page 7 of the paper: Jane: The setup is in place, and page seven delivers the payoff. We already quoted the headline averages, so let's look at what's behind them — starting with the reasoning baselines.

Tom: And there, no single prompting strategy dominates. Chain-of-thought leads on APPS, Self-Planning leads on LiveCodeBench, and CodeAct actually hurts the 7B model on APPS, dropping to 14 point 7 percent versus 16 point 9 for the base model. That's a clean demonstration that giving a model tools without training can break its synthesis ability.

Lu: Against those prompting methods, DiDPO wins by 5 point 4 on APPS and 12 point 8 on LiveCodeBench with the 7B backbone. That gap shows the RL training is providing something prompting alone can't reach.

Meng: The attribution argument against GRPO is what I find cleanest. DiDPO improves over GRPO by 5 point 6 on average with the 7B model, and since both share the same episode-level advantage, that gain comes directly from the sub-diff credit.

Jane: And versus GiGPO, the lead is 4 point 2 on average, with the biggest margin on APPS Interview, at plus 10 point 4, where multi-step reasoning spans multiple functions. The paper explains it as GiGPO grouping by identical environment states, which misses functionally analogous edits in different code regions.

Tom: The second table covers competition-level benchmarks — USACO with its Bronze, Silver, Gold, and Platinum tiers, plus OJBench and ICPC. The standout is USACO at 15 point 6 percent with the 7B model, more than double GRPO's 6 point 8.

Lu: And the SFT baseline underperforms every RL method, which supports their pipeline choice. The SFT stage handles format alignment; the RL stage is what actually lifts reasoning and editing behavior.

Lalam: The pattern across both tables is consistent — the harder and longer the task, the bigger the gain. I suspect that pattern will matter even more as the field moves toward repository-scale tasks with many files.

Meng: Which is exactly what makes the analysis on page eight so interesting, because it shows where those gains come from during training.

Tom: Page eight opens the hood — learning curves, ablations, and where the training time actually goes.

Page 8 of the paper: Tom: Page eight starts with learning dynamics, and the curves tell a clear story. For the first twenty training steps, every method improves at about the same rate, because they all share the episode-level signal.

Jane: After step forty, DiDPO keeps climbing while GiGPO plateaus. The authors read that as the sub-diff credit becoming more informative as the policy diversifies its edits — early on, the edits are too homogeneous to group; later, there's enough variety to compare.

Lu: The group composition analysis reinforces that. They had GPT-5 point 5 classify the sub-diffs into functional blocks, fragments, scaffolds, and other, and over training the functional blocks steadily gain share while fragments and scaffolds decline. The credit assignment is literally reshaping what the policy edits.

Meng: The ablations are the most instructive part. Removing the episode-level signal collapses performance on APPS from 31 point 3 to 10 point 4, so the local signal can't stand alone. Removing the diff-level signal drops to 23 point 8, essentially GRPO-level, and removing the sub-diff decomposition lands at 25 point 0.

Tom: The groupability score design is ablated too. The saturating product form gets 31 point 3, the additive variant gets 24 point 4, and an LLM judge that groups sub-diffs directly gets only 21 point 5 while costing more. That's a strong argument against the obvious alternative of just asking a big model to do the grouping.

Jane: The lambda sensitivity has an inverted-U shape with a peak at 1 point 2, and above that the policy starts overfitting to local credit — learning edits that resemble high-reward peers but don't compose globally. So the two-level combination is doing real work.

Lu: And the efficiency analysis puts the whole thing in perspective. You remember the two percent overhead we mentioned; the breakdown shows it's dominated by the cross-rollout similarity computation, bounded in practice by the similarity threshold, while the greedy anchor selection is linear and nearly free.

Meng: Inference is completely unchanged, since the grouping only happens during training. That's a nice property for anyone thinking about deploying this.

Lalam: So the conclusion frames it modestly — a practical step toward coding agentic RL, with code diffs as the substrate. No grand claims, just a working mechanism.

Tom: So let's pull back now and think about what this paper actually changes for the field.

Conclusion: ident: You're back with the arXiv review channel.

Tom: Before we say goodbye to this paper, let's pull the threads together. What we've seen is a method that moves credit assignment for coding agents down from the whole trajectory to the level of sub-diffs, and it does that without changing the rollout budget.

Jane: The empirical story is consistent. DiDPO beats GRPO and GiGPO on average with both backbones, the gains grow on the harder competition benchmarks, and the cost stays around two percent.

Lu: The ablations tell you which pieces matter. The episode-level signal is the backbone, the diff-level signal adds the fine structure, and the sub-diff decomposition is what makes grouping work at all.

Meng: For people working on agent RL, the open codebase is probably the most immediately useful piece. You can take verl-code, apply DiDPO to your own benchmark, and see whether the groupability ideas transfer to your setting.

Lalam: Stepping back, the interesting shift is that reward structure has become a design choice. The field has moved from sparse outcome rewards to group-relative advantages, and now to advantages organized around the internal shape of what the agent produced.

Tom: That's a fair way to place the paper. The authors also give you a theoretical vocabulary — anchor quality limits bias, group mass limits variance — which I think will outlive the specific algorithm.

Jane: And the frontier comparison should stay in our heads, too. A seven-billion-parameter model trained this way closed roughly a quarter of the gap to GPT-5 point 5. That's the kind of result that makes people rethink how much RL can squeeze out of smaller models.

Lu: The open questions are real, though — how the anchors behave on repository-scale edits, how sensitive the method is to the similarity threshold, whether the grouping could be learned instead of computed.

Meng: But those are questions this paper makes it possible to ask, because it gives you a concrete baseline and the code to build from.

Tom: Exactly. And on that note, we're done with this one. Thanks for listening, and we'll be back with the next paper soon.

Jane: See you all then.

Xucong Wang, Zhe Zhao, Liheng Yu, Di Wu, Xiaofeng Cao, Pengkun Wang

University of Science and Technology of China · Stanford University · Suzhou Institute for Advanced Research, University of Science and Technology of China · Tongji University

cs.AI

Submitted: 2026-08-07

Updated: 2026-08-10

Comments: 16 pages, 6 figures, work in progress

Code: https://github.com/xuc865/verl-code

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 77/100

The gist: Paper Title: DiDPO: Diff-in-Diff Policy Optimization for Coding Agent Training Authors: Xucong Wang, Zhe Zhao, Liheng Yu, Di Wu, Xiaofeng Cao, Pengkun Wang (USTC, Stanford, Suzhou Institute for

Key concepts

Credit assignment: In reinforcement learning, credit assignment is figuring out which actions led to a reward. For coding agents, a single action can change multiple parts of code, but the reward only reflects the final test result, making it hard to know which edits helped.
Diff-in-diff: The core idea of DiDPO: instead of treating a whole code change (diff) as one unit, it splits it into smaller sub-diffs. Similar sub-diffs across different attempts are grouped, and each group gets its own credit signal, providing finer-grained training feedback.
Group-based policy optimization: A family of RL algorithms like GRPO that estimate advantages by comparing multiple rollouts (attempts) of the same task. DiDPO extends this by grouping at the sub-diff level rather than the whole trajectory or state, improving credit assignment without needing a learned critic.

Terminology

Benchmark — 23× in this paper · explained in 2 episodes
GRPO — 21× in this paper · explained in 6 episodes
Groupability Score — 16× in this paper
Reinforcement Learning — 15× in this paper · explained in 4 episodes
Credit assignment — 12× in this paper · explained in 3 episodes
Diff-in-Diff Policy Optimization — 7× in this paper
Ablation — 6× in this paper · explained in 1 episode
Alignment — 6× in this paper · explained in 1 episode
Reinforcement Learning with Verifiable Reward — 5× in this paper
Accuracy — 4× in this paper · explained in 1 episode
LLM judge — 3× in this paper · explained in 2 episodes
Verification — 3× in this paper · explained in 2 episodes
Distillation — 2× in this paper · explained in 2 episodes
Verifier — 2× in this paper · explained in 2 episodes
AI Agents — 1× in this paper · explained in 2 episodes
Ablation Studies — 1× in this paper · explained in 1 episode
Ablation study — 1× in this paper · explained in 1 episode
Actor — 1× in this paper · explained in 3 episodes
Adam — 1× in this paper · explained in 2 episodes
Agentic Task — 1× in this paper · explained in 1 episode
Agentic Tasks — 1× in this paper · explained in 1 episode
Chain-of-Thought Prompting — 1× in this paper · explained in 2 episodes
Core — 1× in this paper · explained in 2 episodes
Deep Reinforcement Learning — 1× in this paper · explained in 2 episodes
Fine-Tuning — 1× in this paper · explained in 5 episodes
Foundation Model — 1× in this paper · explained in 3 episodes
Foundation Models — 1× in this paper · explained in 7 episodes
Generative Model — 1× in this paper · explained in 2 episodes
International Conference on Software Engineering — 1× in this paper
Knowledge Distillation — 1× in this paper · explained in 11 episodes
Redundancy — 1× in this paper · explained in 2 episodes
Supervised Fine-Tuning — 1× in this paper
Supervised Fine-Tuning (SFT) — 1× in this paper · explained in 8 episodes
University of Science and Technology of China — 1× in this paper
Verifiable Rewards — 1× in this paper · explained in 2 episodes
Wasserstein Distance — 1× in this paper · explained in 4 episodes

Summary

Paper Title: DiDPO: Diff-in-Diff Policy Optimization for Coding Agent Training

Core Problem:

Reinforcement Learning with Verifiable Reward (RLVR) has emerged as a powerful paradigm for training coding agents, where execution feedback from compilation and tests provides objective verification. However, the paper identifies a unique, finer-grained credit assignment challenge for coding agents: at each step, coding actions simultaneously pack varying changes into different regions of a code version, which makes the contribution of independent change indistinguishable. Existing RLVR methods mostly leverage outcome rewards or step-level rewards, which fails to dive into a code diff and makes unique properties of coding actions invisible to training.

The paper identifies three structural properties that distinguish coding from existing agentic RL settings:

Bundled snippets: coding actions are bundled with snippets which expand with task difficulty and project scale, meaning harder tasks require longer edits and admit interleaved actions on interleaved regions.
Divisible diffs: "At each step, each diff (i.e., code changes between two adjacent versions) may contain several sub-diffs (i.e., part of the code diffs) with different functionalities. Treating the whole diff as one therefore obscures the semantic unit that produced an improvement."
Strict syntax and pass/fail criteria: code is also governed by strict syntax, execution, and pass/fail criteria; any small malformed edit may influence a long trajectory thus merits special focus.

This leads to the motivating question: How can we construct dynamic, finer-grained credit units for code diffs to better apply agentic RL to coding tasks?

Proposed Method — DiDPO (Diff-in-Diff Policy Optimization):

DiDPO is a critic-free RL method that constructs fine-grained credit units directly from the structure of code diffs. It retains a trajectory-level group-relative advantage but further looks inside each code-producing action. The pipeline works as follows:

Discovering code diffs: Across rollouts of the same task, DiDPO organizes multi-turn coding interactions into thought–action steps and discovers code diffs across sampled trajectories.
Dynamic sub-diff anchors: It identifies highly similar sub-diffs between diffs and uses these sub-diffs as anchors: The resulting sub-diffs capture recurring functional regions from diffs of different rollouts. Matching is restricted to the same edit type (add/del/none), and similarity combines token-level lexical matching and embedding similarity.
Groupability Score (GS): Since over-large and over-small anchors affect the quality of grouping, DiDPO designs a groupability score to select anchors that optimally balance semantic scope they could represent and the mass of the group they could aggregate: GS(c) = (1 − e(−L̄(c)))(1 − e(−(n(c)−1))), where L̄(c) measures average anchor size and n(c) counts occurrences.
Anchor selection as submodular maximization: To avoid redundant overlapping anchors, DiDPO solves a cardinality-constrained facility-location form of submodular maximization that greedily adds anchors with the largest marginal increase.
Diff-level advantage: Anchors form advantage groups G(C⋆), and DiDPO computes a local group-relative advantage AD for each sub-diff action, computed as the normalized group-relative return difference. This is combined with the trajectory-level advantage AE, yielding the final token-level advantage: Â(i,l) = AE(τ(i)) + λ·AD(a l), trained with a standard clipped PPO-style objective. No additional critic or extra environment rollouts are needed.

Theoretical Foundations:

The paper provides two theorems justifying the design:

Theorem 4.1: If the local reward contribution is L-Lipschitz and every sub-diff in a DiDPO group is within correspondence error ϵ of the same anchor, then replacing exact code matches with anchor-based matches changes the local group contrast by at most O(Lϵ), showing better anchor alignment gives less biased local credit.
Theorem 4.2: Assuming step return decomposes into a matched sub-diff component plus zero-mean non-causal trajectory noise with variance σξ2, a DiDPO group with m > 1 sub-diffs satisfies MSE(ÂD i) ≤ O(L2ϵ2) + O(σξ2/m), whereas an episode-level broadcast estimator retains an O(σξ2) contamination term for tokens in the same sub-diff. This shows grouping reduces variance through cross-rollout averaging.

Experiments & Results:

Benchmarks: Eight code-generation benchmarks are used: APPS, HumanEval, MBPP, LiveCodeBench, LeetCode, USACO, OJBench, and ICPC.

Training pipeline: Training uses a cold-start stage where medium-difficulty tasks are augmented with GPT-5.5, Qwen3.6-27B generates up to 12-turn rollouts, and rejection sampling collects 3K trajectories for SFT before RL. Training runs 120 epochs with rollout group size 32, λ = 1.2 (diff-level advantage weight), η = 0.8 (similarity threshold), γ = 0.95, and PPO clipping ratio 0.2.

Main Results: DiDPO achieves the highest average on both backbones: 48.4% with Qwen2.5-Coder-7B and 58.6% with Qwen3.5-4B, surpassing the strongest baseline GiGPO by 4.2 and 4.9 respectively. On Qwen2.5-7B-Coder, DiDPO exceeds comparable methods by over 10% and narrows the gap with far larger models, reducing the gap to GPT-5.5 from 56.4% (base model) to 43.3%. On competition benchmarks (USACO), DiDPO achieves 15.6% with the 7B backbone, more than doubling GRPO's 6.8%.

Compared with baselines: DiDPO improves over GRPO by 5.6 on average (7B backbone), directly attributable to sub-diff credit since both share the same episode-level advantage. It leads GiGPO by 4.2, with the largest advantage on APPS Interview (+10.4), where multi-step reasoning spans multiple functions. Untrained tool-use can disrupt synthesis (CodeAct underperforms base model on APPS: 14.7% vs. 16.9%).

Ablations and Analysis:

Learning dynamics: All methods improve similarly in the first 20 steps; after step 40, DiDPO continues climbing while GiGPO plateaus, indicating that sub-diff credit becomes more informative as the policy diversifies its edits. Group composition analysis shows Block groups (function bodies, loop structures) increase share while Fragment and Scaffold groups decline, matching the DiDPO design where functional blocks earn higher GS.

Component ablations: Removing AE drops APPS from 31.3% to 10.4% (episode-level signal is indispensable); removing AD reduces to 23.8% (isolating a 7.5-point marginal gain from localized credit); removing sub-diff decomposition (treating whole diffs as atomic units) yields 25.0%, confirming that without sub-diff decomposition, partially overlapping diffs cannot be matched, producing smaller and noisier groups.

GS design ablations: The proposed multiplicative saturating exponential form achieves 31.3% on APPS versus 24.4% for an additive variant and 21.5% for an LLM judge (Qwen3.6-27B based), since the additive form allows one strong factor to compensate for a weak one, but the saturating exponential compresses both factors into (0,1), preventing either from dominating.

Sensitivity to λ: Performance follows an inverted U-shape peaking near λ = 1.2; below 0.6, the diff-level signal is suppressed and DiDPO reverts to GRPO-level performance while above 1.2, the policy overfits to local credit.

Efficiency: DiDPO adds only 2.3% overhead over GRPO in per-step training time (dominated by cross-rollout similarity computation), and this overhead is training-only since inference uses standard autoregressive decoding.

Contributions:

The paper's threefold contribution is: (1) identifying three structural properties distinguishing coding from existing agentic RL and formulating the need for hierarchical credit assignment within code-producing actions; (2) proposing DiDPO as the first agentic RL algorithm tailored to the functional hierarchy of code, dynamically aligning diffs and assigning group-relative advantages without extra rollouts; and (3) conducting extensive experiments showing DiDPO consistently outperforms strong policy-optimization baselines. The authors also open-source verl-code, an agentic rl codebase that supports various RL methods and coding benchmarks (https://github.com/xuc865/verl-code).

Improvements for AI systems

Below are the concrete improvements to AI systems enabled by this paper, followed by what the improved system can actually do.

1. Replace coarse trajectory/step-level credit with sub-diff credit

Instead of treating each coding action (one diff) as an atomic unit, decompose it into functional sub-diffs (e.g., a loop restructure, a new helper function, a condition fix).
Group sub-diffs with the same semantic role across different rollouts of the same task, using anchor-based matching with token-level lexical similarity plus embedding similarity.
Compute a local group-relative advantage for each sub-diff and combine it with the episode-level advantage: Â = AE + λ·AD.

2. Add dynamic, matched anchor selection

Use a groupability score GS = (1−e(−L̄))(1−e(−(n−1))) to reject anchors that are too large (low semantic precision) or too rare (low aggregation mass).
Select anchors via greedy submodular maximization (facility-location form) to avoid redundant overlapping anchors.

3. Maintain critic-free, rollout-free training

DiDPO requires no value network and no additional environment rollouts; local advantages are estimated from existing trajectories. This keeps training compute nearly unchanged.

4. Use the open-sourced verl-code framework

Adopt a unified agentic RL codebase supporting multiple RL algorithms and eight coding benchmarks, enabling direct reproduction and extension.

Achieve state-of-the-art code-generation accuracy with small backbones:

48.4% average pass rate with Qwen2.5-Coder-7B and 58.6% with Qwen3.5-4B, beating the strongest baseline (GiGPO) by 4.2 and 4.9 points respectively.
On the 7B backbone, outperform GRPO by 5.6 points on average and narrow the gap to GPT-5.5 from 56.4% down to 43.3%.

Solve competition-level programming problems that stymie prior methods:

On USACO, reach 15.6% with a 7B model—more than double GRPO's 6.8%.

Handle long, multi-function reasoning tasks:

Gain +10.4 points over GiGPO on APPS Interview, where solutions span multiple functions, thanks to the discovery of recurring functional blocks across rollouts.

Keep improving after other methods plateau:

Whereas GRPO and GiGPO flatten after 40 training steps, the sub-diff signal continues to drive gains because the policy progressively diversifies its edits, creating matchable block-level structures.

Provide interpretable training signal:

The system can tell which specific code hunks contributed to a reward change, not just which turn or trajectory did—useful for debugging, reward shaping, and explaining agent behavior.

Train efficiently:

At only 2.3% extra per-step training cost over standard GRPO, the system gets these gains essentially for free; inference-time cost is unchanged since it uses standard autoregressive decoding.

Avoid harmful tool-use interference:

The approach prevents degradation from sub-optimal tool usage (e.g., CodeAct dropping 2.2 points on APPS), because credit is assigned to the actual code changes rather than to the tool-call step that produced them.