S cubed-R1: Learning to Retrieve and Answer Step-by-Step with Synthetic Data
cs.LG
Submitted: 2026-05-02
Updated: 2026-09-06
Comments: Under Review
License: http://creativecommons.org/licenses/by/4.0/
The gist: Reinforcement learning (RL) post-training has enabled newer capabilities in models, such as agentic tool-use for search.
Terminology
Abstract
Reinforcement learning (RL) post-training has enabled newer capabilities in models, such as agentic tool-use for search. However, these models struggle primarily due to limitations with sparse outcome-based rewards and a lack of training data that encapsulates questions of differing hardness, which results in models not performing deeper searches with tools to collect evidence for question-answering. To address these limitations, we introduce S 3-R1 (Synthetic data and stabilized Search R1), a framework that couples a data-centric approach with denser learning signals. We first develop a synthetic generation and curation pipeline that programmatically derives diverse, multi-hop questions from existing documents. This pipeline incorporates a retrieval-based verification step to specifically isolate questions of intermediate difficulty. We then pair this expanded training set with a reward structure that evaluates both intermediate search quality and the correctness of the final answer. This setup directly mitigates the credit assignment problems inherent to sparse rewards. Our evaluations show that S 3-R1 outperforms existing baselines by learning more effective search and synthesis strategies, yielding up to a 10% improvement in robust generalization on out-of-domain datasets.
Sources
- Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs
- The Importance of Pessimism in Fixed-Dataset Policy Optimization
- Self-Questioning Language Models
- Contextualizing Search Queries In-Context Learning for Conversational Rewriting with LLMs
- Retrieval-Augmented Generation for Large Language Models: A Survey
- Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Use
- Reinforced Self-Training (ReST) for Language Modeling
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps
- Grounding by Trying: LLMs with Reinforcement Learning-Enhanced Retrieval
- An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agents
- Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
- PRewrite: Prompt Rewriting with Reinforcement Learning
- Synthetic Data Generation Using Large Language Models: Advances in Text and Code
- MAIN-RAG: Multi-Agent Filtering Retrieval-Augmented Generation
- Proximal Policy Optimization Algorithms
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models
- R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning
- Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks