Cross-Behavior Policy Optimization: Unified Learning from Offline Demonstrations and On- and Off-Policy Rollouts via a Shared Regression Target

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Cross-Behavior Policy Optimization".

Jane: As an excellent, fastidious, and diligent AI researcher, I have meticulously analyzed the provided text excerpt from arXiv.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper right now called Cross-Behavior Policy Optimization: Unified Learning from Offline Demonstrations and On- and Off-Policy Rollouts via a Shared Regression Target. It’s about how we use all our existing data, both the offline demonstrations and the rollouts we generate ourselves, in one go.

Jane: That sounds like it could be a big deal because right now, they seem to be separate things—you do supervised fine-tuning first and then reinforcement learning later—which means you can't really reuse all that collected data easily.

Lu: Exactly. The paper proposes this idea where they make these sources compatible through one regression loss using a fixed reference target. They claim that because the generating policy doesn't enter this target, positive offline completions, negative ones, reference responses, and current and historical rollouts can share a common record format and replay buffer without needing to coordinate separate SFT and RL objectives.

Meng: I’m thinking about what this means for deployment. If you can just throw everything into one buffer and optimize with the same objective, does that make the training pipeline way simpler? Or does it introduce some weird instability when we're running things at scale?

Lalam: Based on what I see in this paper, CBPO-Unified combines offline demonstrations, on-policy rollouts, and off-policy rollouts into one optimization loop using a shared regression target. This means we don't have to store separate behavior policy probabilities or importance weighting anymore.

Tom: Right, so the core idea is that they create this fixed reference target that doesn't depend on the behavior policy itself. They define it as a combination of the completion’s reference-policy log-probability, scaled verifier reward, and a frozen prompt-level normalizer.

Jane: That sounds like they’re trying to anchor the optimization around something stable while still steering toward higher rewards based on that verifier reward. What is the actual math they are using for this target?

Lu: They define it mathematically as "bt(x, y) equals log pi ref of y given x plus r(x, y) beta minus log Z ref of x". And then they fit the model's log-probability to that target using a response-level loss. That structure is what keeps the objective consistent across all data regimes.

Meng: If it keeps the objective consistent, does that mean we don't have to worry about one data type overpowering another when we mix them? I worry about cold starts and having the wrong kind of feedback during early training phases.

Lalam: The advantage they point out is that this allows optimization to retain the same regression objective without coordinating separate SFT and RL objectives or requiring stored behavior-policy probabilities, importance weighting, or a learned critic. It simplifies the coordination aspect significantly.

Paper summary: Tom: They also show variants: CBPO-Offline, CBPO-OnPolicy, CBPO-OffPolicy, and finally CBPO-Unified. This unified version is what they are testing to see if it truly combines everything effectively into one process.

Jane: And looking at the results on Qwen3 model scales and four mathematical reasoning benchmarks, they claim this approach delivers leading average success rates for both avg@thirty-two and pass@thirty-two across all those different data regimes. That’s a pretty broad claim across different setups.

Lu: They show that on Qwen3-1 point 7B, CBPO-Unified improves benchmark-averaged avg@thirty-two by two point three three percentage points and pass@thirty-two by four point four four percentage points over the SFT DAPO baseline, which is a significant lift compared to the previous method.

Meng: Those are solid numbers for a model of that size, but what about scaling up? If you take Qwen3-8B, does this advantage hold up as much when we're dealing with larger models that eat more compute?

Lalam: The paper shows that on Qwen3-8B, the unified approach leads all eight reported benchmark-metric combinations for that scale. Plus, when they looked at historical rollouts specifically for CBPO-Unified, retaining those earlier rollout rounds improved metrics by one point four nine percentage points and two point zero seven percentage points over the no-history configuration.

Tom: So, if we look at the cost side of things, I need to know if this is just a theoretical win or if it's actually practical for production environments where we have to run models on real traffic.

Meng: That’s my main concern. They say on Qwen3-1 point 7B, CBPO-Unified requires one hundred nineteen point three six GPU hours, which they claim is within two percent of DAPO, REINFORCE++, GSPO, and TBRM. And for the Qwen3-8B model, it takes four hundred thirty-seven point four four GPU hours, which they put below non-PPO on-policy baselines and less than half the cost of PPO.

Jane: It’s interesting that they compare it against PPO costs because PPO usually requires a lot of coordination between policy and value functions, which is what CBPO seems to skip by using this shared target. It sounds like the efficiency comes from not having to store or coordinate those extra components.

Lu: The paper also points out the fragmentation in conventional post-training pipelines, where SFT and RLVR are separate objectives that exclude negative demonstrations from SFT and retire positive demonstrations once RLVR begins. CBPO directly addresses that fragmentation by creating a shared regression target for all three data regimes.

Tom: So to sum up the main point of this paper, Cross-Behavior Policy Optimization is a framework that unifies learning from offline demonstrations and on- and off-policy rollouts through one regression loss using a fixed reference target. This makes these sources compatible without needing separate SFT and RL objectives or storing complex policy probabilities.

Jane: And what does that mean for the bigger picture? If we can combine these data streams this smoothly, it suggests a way to leverage much more of the data we’ve already collected before we start training our final policies.

Paper summary: Lalam: It means we can treat positive and negative offline completions, reference responses, and current or historical rollouts as part of one unified record for optimization. This makes the learning process much more holistic across different data types.

Meng: If it works on this scale, it suggests that the way we structure our training data—mixing demonstrations with self-generated data—can be much more efficient than sticking to a strict two-stage pipeline where you have to discard or switch objectives midway.

Tom: The authors are showing that by restricting the available sources to offline, on-policy, and off-policy variants, they can see how this works before moving to the full unified version. It’s a methodical way of proving it step by step.

Jane: They also flag a limitation in their work—the paper states that conventional reasoning post-training commonly follows a two-stage pipeline where SFT is followed by RLVR, and CBPO directly addresses how that fragmentation happens, but they don't necessarily claim to fix every single data usage problem out there.

Lu: That’s right; the limitation they state is about what their method doesn't do, which is that it doesn't necessarily replace every existing post-training setup in the field with this single framework. It’s a specialized unification of existing techniques.

Tom: So, while CBPO-Unified leads all reported benchmark metrics and shows strong efficiency gains compared to SFT DAPO—gaining two point three three percentage points in avg@thirty-two on Qwen3-1 point 7B—the authors are being careful not to overstate its general applicability across every single post-training setup for reasoning.

Jane: So what does this imply for people who only listen to the show? It means that if you’re working with mixed data sources, like having both good offline examples and ongoing rollouts, you might have a more direct path to better performance by unifying your objective instead of keeping them in separate loops.

Lu: The implication is that the structure of the loss function—the fixed reference target—is what makes this work across these different data regimes together. It’s about finding a single mathematical way to guide the policy toward high rewards regardless of whether the data came from an offline demonstration or a current rollout.

Meng: For someone shipping models, it means less plumbing to manage when you want to combine older data with new generation cycles, because the replay buffer handles it without needing complex coordination between separate SFT and RL objectives.

Lalam: I’m confident in what the paper shows about their unified model, as it demonstrates strong results across different model scales and benchmarks. The result is that CBPO-Unified is empirically shown to be the most effective configuration presented in this study.

Conclusion: Tom: So, Cross-Behavior Policy Optimization is this new framework that tries to make offline data and online rollouts work together using one single loss function based on a fixed reference target.

Jane: That’s right, and it uses a shared regression target, which is the mathematical anchor that lets all those different types of data share the same objective.

Lu: The authors focus on unifying these sources so you don't have to coordinate separate supervised fine-tuning and reinforcement learning objectives anymore.

Meng: So what does this mean for deployment? Does it just make the training pipeline smoother or is there some hidden complexity when we try to run this on real traffic?

Lalam: It means you get a common reward-labeled record format for everything, which simplifies the replay buffer and removes the need to store behavior policy probabilities or importance weighting.

Tom: I’m looking at the title of this paper, Cross-Behavior Policy Optimization: Unified Learning from Offline Demonstrations and On- and Off-Policy Rollouts via a Shared Regression Target. It sounds like they are tackling that fragmentation head-on.

Jane: Yeah, it's about taking those two separate learning paths—offline demonstrations and policy rollouts—and finding a single way to train the model using one regression loss.

Lu: The authors show this unification across offline, on-policy, and off-policy variants before they show the full unified version. It’s a methodical approach.

Meng: And looking at the results they show, it looks like CBPO-Unified is the most effective configuration because it combines all those data sources.

Tom: On Qwen3-one point 7B, this unified version improved benchmark-averaged avg@thirty-two by two point three three percentage points and pass@thirty-two by four point four four percentage points over the SFT DAPO baseline. That's a real lift for a smaller model, so that’s interesting to me as an engineer.

Jane: It really shows how much better performance can be when you are able to pull in both old data and new generation cycles without having to switch objectives midway through the training process.

Lu: They also show that retaining historical rollouts specifically improves metrics by one point four nine percentage points on Qwen3-8B for the unified version, which is a good result when you're talking about model scales.

Meng: The cost data they provided is pretty telling too; on Qwen3-8B, it took four hundred thirty-seven point four four GPU hours for the unified approach, which they positioned as being less than half the cost of PPO. That’s something I can actually think about in terms of what we can afford to run at scale.

Tom: So if you only listen to this show, remember this: CBPO is a framework that unifies learning from offline demonstrations and on- and off-policy rollouts through one regression loss using a fixed reference target.

Jane: It’s about creating a single mathematical way to guide the policy toward high rewards regardless of whether the data came from an offline demonstration or a current rollout.

Lu: The implication is that you can treat positive and negative offline completions, reference responses, and historical rollouts as part of one unified record for optimization.

Meng: It means less plumbing to manage when you want to combine older data with new generation cycles because the replay buffer handles it without needing complex coordination between separate SFT and RL objectives.

Tom: This suggests that by structuring your training data this way, you can be much more efficient than sticking to a strict two-stage pipeline where you have to discard or switch objectives midway.

Jane: The authors are being careful not to claim this replaces every existing post-training setup in the field with this single framework; it's a specialized unification of existing techniques.

Lu: They also flag that the method doesn't necessarily fix every single data usage problem out there, which is a fair limitation to acknowledge.

Tom: So while CBPO-Unified leads all reported benchmark metrics and shows strong efficiency gains compared to SFT DAPO—gaining two point three three percentage points in avg@thirty-two on Qwen3-one point 7B—the authors are being careful not to overstate its general applicability across every single post-training setup for reasoning.

Jane: So what this implies for people working with mixed data sources is that you might have a more direct path to better performance by unifying your objective instead of keeping them in separate loops.

Lu: The structure of the loss function—the fixed reference target—is what makes this work across these different data regimes together. It’s about finding a single mathematical way to guide the policy toward high rewards regardless of the data source.