Same Answer, New Box: How Much of RLVR's Out-of-Domain Gain Is Answer Format?

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Same Answer, New Box".

Jane: The gist The gain reported by extractors copied from public code arising from RLVR training disappears under ours,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: We're moving on to how they suggest we improve this measurement. The paper calls these improvements suggestions for future work or for better ways to quantify what we're seeing here.

Jane: It seems they are pushing for a more rigorous way to separate the gain in answers from the gain in format, rather than just reporting a single percentage figure >

Lu: One suggestion is to improve reward-based accuracy reporting by explicitly separating "format gain" from "answer gain." They propose quantifying this by comparing scores generated with extractors that look for specific formatting, like a boxed answer, versus those that read the full text >

Meng: That makes sense if we want to know if the model is actually better at solving the problem or just better at following instructions on where to write it >

Lalam: Another suggestion they have is to implement an extractor-aware scoring mechanism. This would let us report how much of the gain reported by extractors copied from public code arises this way by using re-scoring stored outputs with a second extractor >

Tom: That sounds like a practical way to recover that format gain without having to run all the reinforcement learning training over and over again, which is good for efficiency >

Jane: And they also suggest developing a robust confidence metric for out-of-domain gains. This would assess the reliability of reported accuracy gains by measuring how much those gains disappear when using extractors that look in specific locations >

Lu: That ties back to their finding that eighty-two percent of the gain disappears under their method, which gives us a way to flag when a reported gain might be mostly format >

Meng: I like the idea of flagging gains that are mostly format because it helps us focus on where we need to improve things in the model training itself >

Lalam: They also suggest designing a flexible answer extraction pipeline to adapt to different prompting strategies. This would optimize our output format by dynamically choosing between prompts, since they found the share is seventy-nine percent under the boxed-instruction prompt and eighty-six percent under the plain one >

Tom: Tailoring the reporting based on how you ask the model is a useful thing, especially if we want to see which prompting style leads to better answer quality versus just better formatting >

The paper's summary: Jane: So let's go back to what they actually found about this in the paper called Same Answer, New Box: How Much of RLVR's Out-of-Domain Gain Is Answer Format? They are trying to figure out if the boost from reinforcement learning comes from better reasoning or just better formatting >

Lu: The summary of their main finding is that most of the out-of-domain multiple-choice gain reported by harness extractors disappears under their own method, meaning it's a gain in answer format, not an improvement in answers >

Tom: So to put it plainly, they found that when you look at how much models get better on tasks outside their training range, most of that perceived improvement is actually just the model moving its text into a box where the extractor can find it >

Jane: That means that for many out-of-domain benchmarks, the reported gains are not due to better answers but rather a change in answer format >

Meng: It’s like if you train a car to drive faster, but all you see is that it’s now driving on a different kind of road >

Lalam: They highlight that this happens because base models state most of their answers in text, and the RL models write their answers into boxes >

Tom: So the mechanism is pretty clear: RL training seems to be scaffolding the answer structure for extraction rather than fundamentally improving the underlying knowledge >

Jane: It’s important to note they are looking at this on four released 7B SimpleRL base–RL pairs, and this finding isn't necessarily true for every single model or task > <ref:2610.20639#pg1,on four released 7B SimpleRL base–RL pairs>

Lu: They also found that in in-domain free-form mathematics, where both models already box their answers, an extractor that only reads the boxed answer shows a format gain close to zero >

Tom: So they are saying you can tell the difference between what transfers and what's just structural changes depending on the context of the task >

The paper's improvements: Jane: Now let's talk about those suggestions again. They’re trying to build a system that is more accurate in its reporting. The authors suggest ways to improve how we measure this format gain >

Lu: They suggest the first improvement is separating format gain from answer gain by comparing scores from extractors that look for specific formatting versus those that read the full text, which they call measuring how much of the gain reported by extractors copied from public code arises this way >

Meng: That sounds like a way to quantify it, not just say it's mostly format; we need a number for what we're losing when we only look at boxes >

Lalam: They also suggest an extractor-aware scoring mechanism. This would be doing that by re-scoring stored outputs with a second extractor, which they say recovers the format gain without needing to run RL again >

Tom: That’s smart because it means we don't have to redo all the training just to get a better picture of the format effect >

Jane: They also want a more robust confidence metric for out-of-domain gains. This would check reliability by measuring how much those gains disappear when using extractors that look in specific locations >

Lu: That links back to their finding that eighty-two percent of the gain disappears under their method, so it gives us a statistical measure of how much the reported gain might be noise >

Meng: I see this as a way to build trust in the numbers we see when we look at these benchmarks >

Lalam: Finally, they suggest designing a flexible answer extraction pipeline to adapt to different prompting strategies. This would optimize our output format by dynamically choosing between prompts, since they found the share is seventy-nine percent under the boxed-instruction prompt and eighty-six percent under the plain one >

Tom: So it's about building a tool that helps us report these gains in a way that accounts for how we interact with the model >

Conclusion: Jane: Alright, let’s wrap this up. The main implication of this work is that when we look at RLVR models getting out-of-domain gains, a large part of that credit might actually belong to the program that read the answers >

Lu: So even if the RL process improves reasoning, if it just moves the answer into a box, that credit shifts because the extraction method is what's changing >

Tom: The final thought is to name your extractor behind every reported number and report on how much gain comes from format gain before any RL model exists >

Meng: For deployment at scale, we need to make sure we are measuring what actually matters for real traffic, not just these specific benchmark numbers >

Lalam: And I'm confident that by using this measurement of format gain in papers like Same Answer, New Box: How Much of RLVR's Out-of-Domain Gain Is Answer Format?, we can start separating the structural changes from actual answer quality >

Jane: That’s a clear direction for where this research is going. We have a lot to think about as we look at the next paper on arXiv >