Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners".

Jane: The gist The Thinking-Free Policy Initialization (TFPI) bridges long Chain-of-Thought (CoT) distillation and standard Reinforcement Learning with Verifiable Reward (RLVR) to make distilled reasoning models more effective and efficient Reasoners…

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're looking at this paper, "Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners," and it's about taking a method called RLVR, which is good for complex tasks but uses really long context lengths during training, and adding this Thinking-Free Policy Initialization thing to make it faster.

Jane: That sounds complicated, Tom. Can you tell us what the authors are trying to solve with this idea? What's the main problem they're tackling?

Tom: Basically, RLVR solves hard tasks but it demands extremely long context lengths during training, which means huge computational costs. They say that starting with short contexts causes bad performance degradation, so this TFPI stage bridges long Chain-of-Thought distillation and standard RLVR to fix that.

Lu: The core idea is using a ThinkingFree operation, which is basically explicitly discarding the thinking content by appending a direct </think> tag, to reduce token usage during inference. It's a simple adaptation to RLVR.

Meng: But it's not just about saving tokens for inference; they are training with these ThinkingFree-adapted inputs and showing that this actually improves performance and lowers token consumption even when the model is in its original slow-thinking mode.

Jane: So, if you're training with this modified input, does it help the model keep its slow reasoning ability intact?

Lalam: Yes, it shows that training with ThinkingFree-adapted inputs improves performance and lowers token consumption even when evaluated in the original slow-thinking mode, which is pretty interesting because they show that omitting the thinking content doesn't hurt the reasoning capabilities.

Tom: That’s a clean point. The paper also introduces this TFPI stage before standard RLVR, with the objective being modified to JTFPI(theta) equals Ex∼D

JRLVR(theta, x′): , where x prime is the ThinkingFree version of x.

Lu: During the rollout stage of TFPI, they generate responses conditioned on that thinking-free query x prime, and then they adapt the importance ratios and advantages using a specific formula that involves pi theta old and pi theta new.

Jane: That sounds like a pretty detailed training setup. What are the main results showing about this whole process?

Tom: The paper presents four main contributions. First, ThinkingFree can substantially reduce inference costs for distilled LRMs, with the ThinkingFree operation reducing output tokens by more than seventy percent for both models <ref:2509.26226#pg2>.

Title and authors: Meng: And second, they propose TFPI as a fast, low-cost initialization phase for long Chain-of-Thought RL that accelerates RL convergence and generalizes across domains even when trained solely on mathematics.

Lu: They also show that long Chain-of-Thought RL following TFPI achieves a higher performance ceiling while producing more token-efficient LRMs without needing specialized rewards or complex training designs.

Jane: So, they're saying this route offers a more effective and efficient way to train high-performing models without needing extra reward shaping.

Tom: And fourth, they reveal that TFPI not only preserves the slow-thinking reasoning pattern but also enables substantially faster rollouts in subsequent long Chain-of-Thought RL stages.

Lalam: They found that TFPI preserves the core reasoning pattern of slow-thinking rather than drifting toward an excessively extended “slow-slow thinking” behavior, which is a key detail for preserving that capability.

Tom: Looking at performance, they show TFPI substantially enhances the slow-thinking capabilities of distilled LRMs even when trained with a small response length. For instance, on DS-1 point 5B, TFPI Stage one raises the overall average accuracy from twenty-two point zero percent to twenty-six point seven percent plus four point seven percent despite being restricted to a two thousand token training response length <ref:2509.26226#pg2>.

Jane: That's a significant jump in accuracy for such a small training context, Tom; what does that mean for models that are limited in how much they can learn from their initial data?

Tom: It means they show that this technique helps improve performance and lowers token consumption, even when evaluated in the original slow-thinking mode. Compared to “Direct RL,” Qwen3-4B with TFPI hits sixty-three point eight percent overall accuracy versus sixty point two percent for Direct RL, which is a plus of three point six percent <ref:2509.26226#pg2>.

Meng: From a deployment standpoint, the token efficiency is huge here; they show that the total compute cost of all three TFPI stages amounts to less than twenty percent of standard RL training with thirty-two thousand token sequences.

Lu: And they also see improvements in generalizability across domains, even when trained exclusively on mathematics data like Polaris-53K, demonstrating out-of-domain improvements as well <ref:2509.26226#pg3>.

Jane: So, this suggests TFPI isn't just a niche mathematical trick; it has some broader applicability for improving how we train large language models.

Title and authors: Tom: It offers a cost-efficient intermediate stage between long Chain-of-Thought distillation and standard RL training, and the total compute cost is very low compared to standard RL training with thirty-two thousand token sequences.

Lu: The analysis also shows that TFPI can raise the upper bound of RL-trained performance for Qwen3-4B, boosting AIME25 accuracy from seventy point six percent to seventy six point zero percent plus five point four percent <ref:2509.26226#pg2>.

Jane: This gives us a solid idea that we don't necessarily need highly specialized rewards or complex training designs to get high-performing models with this route.

Lalam: And they found that TFPI consistently lies on the Pareto frontier for accuracy versus token usage, which means you get token-efficient LRMs offering an alternative pathway to train models that deliver both high accuracy and reduced output length.

Tom: So, while it's a complex setup involving three stages—Stage one Stage two and Stage three—the overall compute cost is less than twenty percent of standard RL training with thirty-two thousand token sequences <ref:2509.26226#pg1>.

Lu: And the behavioral analysis reveals that TFPI preserves the slow-thinking reasoning pattern rather than drifting toward an excessively extended “slow-slow thinking” behavior, which is good because it maintains the core pattern.

Jane: It sounds like a very structured approach to making models more effective and efficient, even when you're dealing with those long context challenges.

Tom: So, as we wrap up this discussion on "Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners," TFPI provides a cost-efficient intermediate stage between long Chain-of-Thought distillation and standard RL training.

Lu: It offers an alternative paradigm where a strong slow-thinking LRM can be obtained, and a more efficient variant can be realized simply by switching to the thinking-free mode without any additional length control mechanisms.

Meng: And for production, it's appealing because it bypasses the need for specialized reward shaping by switching to this thinking-free mode.

Jane: It’s a neat way to achieve high accuracy and reduced output length together without needing those extra complex training designs we often have to build in.

Tom: We'll be looking at how this Thinking-Free Policy Initialization works as we move into the next paper on arXiv.

The paper's summary: Tom: So, to summarize what we just heard, this paper introduces something called Thinking-Free Policy Initialization or TFPI, which is essentially a new setup for training large language models that use long chains of thought.

Jane: Right, so instead of just throwing the model into standard reinforcement learning with verifiable reward after distillation, they create this specific starting phase that aims to cut down on rollout costs and boost reasoning power.

Tom: Exactly, it's about modifying the training objective itself so that when you evaluate it later in the rollout stage, you’re using a version of the query where all the thinking content has been explicitly removed.

Lu: The core mechanic is this operation that takes an input and turns it into a thinking-free version, x prime, which is then used to condition the generated responses during that initial phase.

Jane: So, what does this actually do for the model's learning process? Is it just a trick to save tokens later or does it change how the model reasons?

Lalam: It changes how they learn by training with these thinking-free inputs, and they found that this improves performance and lowers token consumption even when you test the model in its original slow-thinking mode.

Tom: That’s a key finding, because it means you can actually train the model to be better while still keeping that slow reasoning pattern intact.

Meng: But how does this translate to what we actually deploy? If I have a model running on real traffic, how much faster is this whole process compared to just doing standard RL training?

Jane: Well, the numbers suggest that TFPI isn't just an academic exercise; it provides a fast, low-cost initialization phase for long chain-of-thought reinforcement learning that actually accelerates convergence and works well even when you only train on pure mathematics data.

Tom: The paper shows this stage can raise the upper bound of performance in standard RL training, which is pretty significant because it means you can get a higher accuracy ceiling with less total compute time.

Lu: And they show that long chain-of-thought reinforcement learning that follows this TFPI structure reaches a higher performance ceiling while producing more token-efficient models without needing any specialized rewards or complex training designs.

Jane: So, the big picture is that this gives us a more effective and efficient route to training high-performing language models, and it bypasses the need for those complicated reward shaping techniques.

Lalam: And behaviorally, they showed that TFPI preserves the slow-thinking reasoning pattern instead of letting it drift into some kind of overly extended thinking behavior.

Tom: The efficiency trade-off is really interesting too; they found that TFPI consistently sits on the frontier for accuracy versus token usage, giving you token-efficient models with high accuracy.

Meng: So, if I'm looking at deployment costs and model size, it sounds like this method makes a lot of sense because the total compute cost across all three stages is less than twenty percent of standard reinforcement learning training with thirty-two thousand token sequences.

Jane: It’s a very structured approach that helps bridge the gap between long chain-of-thought distillation and traditional RL, offering an alternative where you can get a strong slow-thinking model and then optimize it for efficiency.

Tom: And they’re suggesting that for real deployment, you can realize this more efficient variant simply by switching to the thinking-free mode without needing extra length control mechanisms.

Lu: Looking ahead, they suggest that incorporating more diverse training data across different domains might help because the improvements on mathematical benchmarks are consistent across stages, while other domains can fluctuate.

Jane: That points toward a future where this thinking-free verification behavior learned during TFPI could transfer over to inference in the original slow-thinking mode.

The paper's improvements: Tom: So, we've been talking about how this Thinking-Free Policy Initialization stage changes the training process for distilled reasoning models, and now we're looking at what specific improvements they claim they get out of it.

Jane: Right, so beyond just saving tokens during inference and boosting accuracy a little bit, what are the actual performance gains they are seeing in concrete numbers?

Tom: The first big thing is that this method substantially enhances the slow-thinking capabilities of distilled models even when you're only training them with a very small response length.

Lu: They show that for instance, on a DS-one point 5B model, applying TFPI Stage one raises the overall average accuracy from twenty-two point zero percent up to twenty-six point seven percent, and they did this despite being restricted to only two thousand token training responses <ref:2509.26226#pg2>.

Jane: That's a significant accuracy jump for such a small training context; what does that mean for models that are limited in how much they can learn from their initial data?

Tom: It means the technique helps improve performance and lowers token consumption even when you evaluate the model in its original slow-thinking mode, which is pretty telling.

Meng: From a deployment standpoint, if I'm looking at running these models on real traffic, how much faster is this whole process compared to just doing standard RL training?

Jane: Compared to "Direct RL," Qwen3-4B with TFPI hits sixty-three point eight percent overall accuracy versus sixty point two percent for the direct method, which is a gain of three point six percent <ref:2509.26226#pg2>.

Tom: And when you look at the resource cost, they found that the total compute cost across all three TFPI stages amounts to less than twenty percent of standard RL training with thirty-two thousand token sequences.

Lu: They also demonstrate that this improvement has some degree of generalizability across different domains, even though they trained it exclusively on mathematics data like Polaris-53K, showing out-of-domain improvements too <ref:2509.26226#pg3>.

Jane: So, this suggests that TFPI isn't just a niche mathematical trick; it has broader applicability for improving how we train large language models in general.

Tom: The most exciting part is that they showed TFPI can raise the upper bound of RL-trained performance for Qwen3-4B, boosting AIME25 accuracy from seventy point six percent up to seventy six point zero percent plus five point four percent after applying reinforcement learning <ref:2509.26226#pg2>.

Meng: That's a solid result for scaling up reasoning; it shows we can push the limits of what these models can achieve with less heavy lifting during the training phase itself.

Jane: It really does suggest that you don't necessarily need highly specialized rewards or complex training designs to get high-performing models when you use this route.

Lalam: And behaviorally, they confirmed that TFPI preserves the slow-thinking reasoning pattern instead of letting it drift into some kind of overly extended thinking behavior.

Tom: That preservation is important because it means you keep the core reasoning pattern intact, which is what we want from these distilled models.

Lu: The analysis also revealed that TFPI enables substantially faster rollouts in subsequent long chain-of-thought reinforcement learning stages, meaning you get to the final answer quicker.

Jane: So, while it's a structured approach involving three stages—Stage one Stage two and Stage three—the overall compute cost is low compared to standard RL training with thirty-two thousand token sequences.

Tom: And they found that TFPI consistently lies on the Pareto frontier for accuracy versus token usage, which means you get token-efficient models offering an alternative pathway to train models that deliver both high accuracy and reduced output length.

Meng: It sounds like a very structured approach to making models more effective and efficient, even when you're dealing with those long context challenges.

Jane: Exactly, it’s a neat way to achieve high accuracy and reduced output length together without needing those extra complex training designs we often have to build in. The next paper we're looking at is about how AI models can recover ethnicity from individual records, which is a bit different but touches on how models learn patterns from data.

Conclusion: Tom: So we're wrapping up on "Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners," which is essentially this new way to initialize long chain-of-thought models before they even start their reinforcement learning training.

Jane: It’s been really interesting seeing how they managed to bridge the gap between distillation and standard reinforcement learning in such a cost-effective way for reasoning models.

Tom: Exactly, it’s about creating this fast low-cost initialization phase that speeds up convergence and helps them generalize across different domains even if they only trained on math data.

Lu: I just want to say that the way they preserve the slow thinking pattern rather than letting it drift into some kind of overly extended behavior is a very strong observation for representation learning.

Meng: From a production standpoint, if you're looking at deploying models, this means you can get high performance with less overall compute time and without needing those specialized reward shapes we usually have to design.

Lalam: I’m really confident in my assessment that TFPI provides a solid framework for training strong slow-thinking LLMs efficiently.

Jane: So, the main implication is that we can get a much more effective and efficient route to training high-performing language models without needing those complex training designs.

Tom: And the total compute cost of all three stages amounts to less than twenty percent of standard RL training with thirty-two thousand token sequences, which makes it very attractive for scaling up.

Lu: The future work they mentioned about incorporating more diverse training data across multiple domains is smart because mathematical improvements are consistent, but other domains fluctuate.

Meng: I'm just wondering what the practical limitations are if you try to apply this to something completely different than math problems.

Jane: That’s a good point; the authors flag that while the verification behaviors learned during TFPI can generalize, they're still looking at how those learned verification behaviors transfer to slow-thinking verification occurring in other types of reasoning.

Tom: Yeah, so there's potential for transferring that thinking-free training directly into inference in thinking mode down the road.

Lalam: I’m sure the model is quite sure of this direction; the literature suggests a lot about how these verification behaviors are learned during that initialization.

Jane: Well, that’s all we have time for today on "Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners," but we'll be back next week to talk about how AI models can actually recover ethnicity from individual records.