Future Policy Approximation for Offline Reinforcement Learning Improves Mathematical Reasoning

arXiv:2509.19893 · cs.CL · Submitted 2026-08-19 · Read on arXiv

Minjae Oh, Yunho Choi, Dongmin Choi, Yohan Jo

cs.CL

Submitted: 2026-08-19

Updated: 2026-08-20

Code: https://github.com/huggingface/trl

Terminology

• Benchmark — 19× in this paper · explained in 2 episodes
• Accuracy — 18× in this paper · explained in 1 episode
• Entanglement — 14× in this paper · explained in 4 episodes
• Reinforcement Learning — 14× in this paper · explained in 4 episodes
• GRPO — 12× in this paper · explained in 6 episodes
• Mirror Descent — 12× in this paper
• Regularization — 12× in this paper · explained in 3 episodes
• Alignment — 9× in this paper · explained in 1 episode
• Future Policy Approximation — 9× in this paper
• Direct Preference Optimization — 8× in this paper
• Optimistic Mirror Descent — 8× in this paper
• Ablation — 7× in this paper · explained in 1 episode
• Fine-Tuning — 6× in this paper · explained in 5 episodes
• Direct Preference Optimization (DPO) — 5× in this paper · explained in 9 episodes
• Adam — 4× in this paper · explained in 2 episodes
• Computational overhead — 4× in this paper · explained in 2 episodes
• Loss Function — 4× in this paper · explained in 2 episodes
• Supervised Fine-Tuning — 4× in this paper
• KL Regularization — 3× in this paper · explained in 2 episodes
• Kahneman-Tversky Optimization — 3× in this paper
• Reasoning Preference Optimization — 3× in this paper
• Reinforcement Learning (RL) — 3× in this paper · explained in 25 episodes
• Score Function — 3× in this paper · explained in 4 episodes
• Supervised Fine-Tuning (SFT) — 3× in this paper · explained in 8 episodes
• Bradley-Terry model — 2× in this paper · explained in 2 episodes
• Decoupling — 2× in this paper · explained in 2 episodes
• Foundation Model — 2× in this paper · explained in 3 episodes
• Foundation Models — 2× in this paper · explained in 7 episodes
• Group Relative Policy Optimization — 2× in this paper
• Large Language Models (LLMs) — 2× in this paper · explained in 6 episodes
• Verifiable Rewards — 2× in this paper · explained in 2 episodes
• Algorithmic Improvements — 1× in this paper · explained in 1 episode
• Benchmarking — 1× in this paper · explained in 3 episodes
• Chain-of-Thought (CoT) — 1× in this paper · explained in 4 episodes
• Chain-of-Thought (CoT) Reasoning — 1× in this paper · explained in 2 episodes
• Core — 1× in this paper · explained in 2 episodes
• Distillation — 1× in this paper · explained in 2 episodes
• Failure Modes — 1× in this paper · explained in 3 episodes
• Group Relative Policy Optimization (GRPO) — 1× in this paper · explained in 5 episodes
• Mathematics Aptitude Test of Heuristics — 1× in this paper
• Objective Function — 1× in this paper · explained in 2 episodes
• Perplexity — 1× in this paper · explained in 3 episodes
• Proximal Policy Optimization — 1× in this paper
• Proximal Policy Optimization (PPO) — 1× in this paper · explained in 5 episodes
• Reward Function — 1× in this paper · explained in 2 episodes
• Transformer — 1× in this paper · explained in 2 episodes
• Verifier — 1× in this paper · explained in 2 episodes

Sources

• MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention
• Training Verifiers to Solve Math Word Problems
• On Group Relative Policy Optimization Collapse in Agent Search: The Lazy Likelihood-Displacement
• The Llama 3 Herd of Models
• Reinforced Self-Training (ReST) for Language Modeling
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• Direct Language Model Alignment from Online AI Feedback
• OpenAI o1 System Card
• The Art of Scaling Reinforcement Learning Compute for LLMs
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs
• Tulu 3: Pushing Frontiers in Open Language Model Post-Training
• EXAONE 4.0: Unified Large Language Models Integrating Non-reasoning and Reasoning Modes
• EXAONE Deep: Reasoning Enhanced Language Models
• PCL-Reasoner-V1.5: Advancing Math Reasoning with Offline Reinforcement Learning
• Training language models to follow instructions with human feedback
• Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive
• Trust Region Policy Optimization
• Proximal Policy Optimization Algorithms
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering