Conditional Transfer from Controlled Pretraining Mixtures to Code

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Conditional Transfer from Controlled Pretraining Mixtures to Code".

Jane: The gist The best observed HumanEval pass@20 came from a mixture in which the target-aligned source held 75% of the slice and a highly teachable curated synthetic suite held the…

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we've looked at how synthetic tasks can be used for pretraining or as probes, and this paper is about how mixing those different types of data affects the AI's final ability to write code (Conditional Transfer from Controlled Pretraining Mixtures to Code page one of that work reads: "Ohad Rubin Abstract Synthetic tasks are increasingly used both as probes of language-model capability and as pretraining data").

Jane: It’s called Conditional Transfer from Controlled Pretraining Mixtures to Code. The authors are looking at these mixtures and trying to figure out which ones actually help the model learn coding skills better (Conditional Transfer from Controlled Pretraining Mixtures to Code page one of that work reads: "These two uses are usually justified by the same evidence: a task whose loss falls during training is treated as informative, and a task whose loss falls faster when it is sampled more is treated as worth sampling").

Lu: Essentially, they separate the signals in how loss is going down during training. They track whether a task is diagnostic, if it’s teachable based on its own sampling budget, and if including it actually improves a final coding test score like HumanEval pass at twenty (Conditional Transfer from Controlled Pretraining Mixtures to Code page two of that work reads: "Our contributions are as follows. one. Three separable signals. We give operational definitions of diagnosticity (loss tracks global training progress), teachability (loss responds to the task’s own token budget), and transfer (a source’s share improves the downstream target)").

Meng: The experiment set up fixed seventy percent general Python and varied the remaining thirty percent between instruction-based code, curated synthetic tasks, and literature-derived probes. It’s about finding the right recipe for that mix (Conditional Transfer from Controlled Pretraining Mixtures to Code page three of that work reads: "Every pretraining run draws seventy percent of its tokens from a fixed general Python corpus (Base) from CodeParrot and thirty percent from a controlled slice whose composition is the only manipulated variable").

Lalam: The core finding is that if you want good code transfer, you need to keep the source that's directly related to the target task dominant in your mixture (Conditional Transfer from Controlled Pretraining Mixtures to Code page ten of that work reads: "For code transfer, keep the aligned source dominant").

Tom: So what does this mean for us? It suggests we can't just throw all synthetic data into the model randomly; we have to choose our ingredients carefully (Conditional Transfer from Controlled Pretraining Mixtures to Code page one of that work reads: "Synthetic tasks are increasingly used both as probes of language-model capability and as pretraining data").

Jane: Right. The paper shows that curated synthetic data has value, but only when it’s part of a larger mix where the instruction-based source still holds most of the power (Conditional Transfer from Controlled Pretraining Mixtures to Code page six of that work reads: "Curated synthetic data therefore has conditional value: it contributes as a limited share of a mixture that a target-aligned source still dominates").

Paper summary: Lu: It’s interesting how they found that mixing seven-five percent OpenCodeInstruct with a twenty-five percent curated suite gave the best HumanEval score (Conditional Transfer from Controlled Pretraining Mixtures to Code page seven of that work reads: "On the mixture simplex edge between the curated suite and OpenCodeInstruct, HumanEval pass@twenty after a fixed fine-tuning stage rises from fifteen point nine at pure curated data to its highest observed value, twenty-two point six, at a mixture that is seventy-five percent OpenCodeInstruct").

Meng: That points to complementarity between those two data types, maybe reusable operations and target instructions. It's not just one thing working well alone (Conditional Transfer from Controlled Pretraining Mixtures to Code page six of that work reads: "The interpretation suggests complementarity between reusable program-like operations and target-aligned instruction data").

Tom: And then there’s this adaptive scheduler they tested; it tried to follow the teachability signal but ended up pushing away from the best transfer region for most of the training time (Conditional Transfer from Controlled Pretraining Mixtures to Code page seven of that work reads: "Ado drives the OpenCodeInstruct share below five percent within the first 5k steps and to one point two--one point four percent by the end of training, and underperforms its matched fixed-mixture controls by two point four--eleven point zero percentage points").

Jane: So what does that tell us about how we should tune our training processes? It suggests optimizing for immediate loss reduction isn't always what leads to better final performance on a specific task (Conditional Transfer from Controlled Pretraining Mixtures to Code page seven of that work reads: "Optimizing near-term task-loss reduction moves the mixture away from the region that transfers").

Lu: The conclusion is that for code transfer, keep the aligned source dominant, treat curated synthetic data as a bounded share, and pick proportions based on signals that point toward transfer (Conditional Transfer from Controlled Pretraining Mixtures to Code page ten of that work reads: "For code transfer, keep the aligned source dominant, treat curated synthetic data as a bounded share, and choose proportions with a target-referenced signal").

Meng: So we need to think about the long game of pretraining mixtures rather than just chasing the fastest loss drop right now. That’s what this paper suggests for building better code AI (Conditional Transfer from Controlled Pretraining Mixtures to Code page ten of that work reads: "So we need to think about the long game of pretraining mixtures rather than just chasing the fastest loss drop right now").

Tom: So the big picture here is that understanding how different data sources interact—whether they are diagnostic or teachable—is more important than just seeing which one has the lowest loss right now (Conditional Transfer from Controlled Pretraining Mixtures to Code page one of that work reads: "The authors are defining these three signals very precisely").

Jane: And that's what this paper on Conditional Transfer from Controlled Pretraining Mixtures to Code is all about.

Conclusion: Tom: So we're wrapping up this look at how we mix pretraining data to get better code generation results. The paper is titled "Conditional Transfer from Controlled Pretraining Mixtures to Code".

Jane: That title really tells you what they're doing—it’s not just about using synthetic tasks, it’s about *how* you condition that mixture.

Lu: The authors are defining three ways we can measure these signals: diagnostic, teachability, and transfer. They show that mixing data is complex because those signals don't always line up with what actually helps the final coding test score like HumanEval pass at twenty.

Meng: What this really means for us is that we can't just throw all synthetic data into the model randomly; we have to be strategic about which source families we mix in based on where you are in the training process.

Tom: Exactly. We learned that optimizing for immediate loss reduction, which is what some schedulers try to do, doesn't always lead to better transfer performance for the actual coding task.

Jane: It suggests that chasing every single drop of loss right now might actually push the model away from the sweet spot where it learns how to generate code effectively.

Lu: The implication here is that we need a more nuanced approach when we design these training mixtures if we want to get high-quality code results out of these models.

Meng: For me, it means when I'm building things, I can’t just blindly pile on every piece of synthetic data; I have to think about which source family is most relevant for the final application.

Tom: So the big picture here is that understanding how different data sources interact—whether they are diagnostic or teachable—is more important than just seeing which one has the lowest loss right now.