Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Models
summary
The gist
The gist The authors propose Composition-RL, an approach that automatically composes multiple problems into a new verifiable question and uses these compositional prompts for RL training, which
In short
Composition-RL addresses limited verifiable prompts in Reinforcement Learning by automatically composing existing problems into new, more complex ones before training. This method, Composition-RL, mitigates the issue of easy prompts dominating training data by creating harder compositional prompts. Experiments show this approach consistently improves reasoning capability across various model sizes and benchmarks.
Key concepts
- Composition-RL
- An offline data-augmentation technique that combines multiple existing verifiable prompts into a single, new complex prompt. This is done before the Reinforcement Learning training begins to create more challenging training examples.
- Sequential Prompt Composition (SPC)
- A procedure to build a new prompt by sequentially modifying existing ones. It involves using the ground-truth answer of one prompt to modify another, introducing natural language definitions and relations between variables. This is generalized recursively for many prompts.
- Curriculum RL
- A variant of Composition-RL where the level of compositional complexity gradually increases during the training process. The model starts with simpler compositions and then moves to deeper compositions as performance stabilizes, leading to further gains.
- Cross-Domain Composition
- Using Composition-RL to create training data by combining prompts from different knowledge domains, such as math and physics. Training on this cross-domain dataset improves the model's reasoning not just in the original subjects but also in unrelated fields.
Terminology used across episodes
This episode discusses
- Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Models · Paper Radio
- Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Models
- On Predictability of Reinforcement Learning Dynamics for Large Language Models
- MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention
- Nudging the Boundaries of LLM Reasoning
- MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoning
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- JustRL: Scaling a 1.5B LLM with a Simple RL Recipe
- DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning
- Measuring Mathematical Problem Solving With the MATH Dataset
- Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model
- OpenAI o1 System Card
- Measuring Compositional Generalization: A Comprehensive Method on Realistic Data
- No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping
- QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentation
- InternBootcamp: Boosting LLM Reasoning with Verifiable Task Scaling · Paper Radio
- Knapsack RL: Unlocking Exploration of LLMs via Optimizing Budget Allocation
- DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
- ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
- SAND-Math: Using LLMs to Generate Novel, Difficult and Useful Mathematics Questions and Answers
- MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learning
The paper
Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Models · Read on arXiv
Xin Xu, Clive Bai, Kai Yang, Tianhao Chen, Yangkun Chen, Weijie Liu, Hao Chen, Yang Wang
University of Science and Technology of China
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: Today's paper: "Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Models".
Jane: The gist The authors propose Composition-RL, an approach that automatically composes multiple problems into a new verifiable question and uses these compositional prompts for RL training,
Tom: First, who's behind it and why it matters.
Paper summary: Tom: So, to conclude on Composition-RL: it consistently outperforms RL on the original prompts and shows the potential of composing prompts from different topics >
Jane: The title of the paper, Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Models, really captures what they did here >
Lu: It’s about taking a dataset that has too many easy examples and using a clever composition method to generate more challenging training material for the AI >
Meng: It suggests that instead of just throwing more raw data at the model, we can structure that data so it actively helps the model learn better intermediate steps >
Tom: Yeah, so Composition-RL provides this implicit process supervision by encouraging correct intermediate steps, and they’re releasing their codes and datasets to support future research on RL >
Conclusion: Jane: So, to wrap up this part, Composition-RL is basically about taking existing questions and stitching them together into bigger, more complex prompts for the AI to learn from >
Tom: Yeah, they’re using this technique to fix a problem where the training data gets too easy or redundant when you're doing Reinforcement Learning with Verifiable Rewards >
Jane: And it seems like the authors are calling this specific method Sequential Prompt Composition, or SPC, which is just a fancy way of describing how they build these new questions step by step using the answers from the old ones >
Lu: From my side, I see this as a way to inject structured dependency into the training signal. It’s not just giving the AI one question; it’s giving it a chain of reasoning where each link depends on what came before >
Meng: But for me, what matters is that they're doing this offline augmentation. They generate these composed datasets once and then use them for the RL training, which means you don't have to worry about running complex composition logic during the actual training loop >
Tom: I liked seeing how they showed that composing these prompts actually improves reasoning capability over just using the original dataset, even across different model sizes >
Jane: It makes sense because if you're forcing the AI to bridge a gap between two related concepts, it has to learn a new skill about how those pieces fit together >
Lu: I think the real power they’re showing is in cross-domain RL. They take prompts from one area, like physics, and combine them with prompts from another, like math or law >
Meng: That opens up possibilities for training AI on things it hasn't seen much of directly, which is what we really need for real-world deployment >
Tom: It really highlights that the quality of the prompt data is a huge lever for RL performance when you're dealing with these large models >
Jane: So, even though they show some impressive results on MATH and physics, I wonder how robust this compositional structure is when we move to completely different types of knowledge >
More episodes
- 2606.01198-Linear Strategic Classification with Endogenous Improvements
- 2603.22735-Explanation Generation for Contradiction Reconciliation with LLMs
- 2602.13832-BASIS: Evaluating Belief-Aware Solutions in Interactive Settings beyond Instructions
- 2606.11946-Neuro-Relational Programs: Unifying Queries and Neural Computation over Structured Data
- 2606.28573-Replica Symmetry Breaking and Algorithmic Thresholds in Empirical Risk Minimization under Multi-Index Model
- 2607.01670-UniWind: Day-Ahead Wind Power Forecasting via Physics-Informed State Routing
- 2608.15146-PureTD: Reinforcement Learning for Backgammon Money Games with No Evaluation-time Search
- 2506.01495-C-VARC: A Large-Scale Chinese Value Rule Corpus for Value Alignment of Large Language Models
- 2601.21132-Large Language Models Naively Recover Ethnicity from Individual Records
- 2601.21257-MoCo: A One-Stop Shop for Model Collaboration Research