Iterative GRPO: Batch-Online Multi-Turn RL via Single-Turn RLHF

arXiv:2511.21638 · cs.LG · Submitted 2025-11-26 · Read on arXiv

cs.LG

Submitted: 2025-11-26

Updated: 2026-09-04

Code: https://github.com/facebookresearch/xformers

Terminology

• Suggested Response — 15× in this paper
• Reinforcement Learning — 12× in this paper · explained in 4 episodes
• Action Space — 3× in this paper · explained in 1 episode
• Adaptability — 3× in this paper · explained in 1 episode
• Large Language Models (LLMs) — 3× in this paper · explained in 6 episodes
• Q-learning — 3× in this paper · explained in 3 episodes
• Reward Function — 3× in this paper · explained in 2 episodes
• Deep Reinforcement Learning — 2× in this paper · explained in 2 episodes
• Fine-Tuning — 2× in this paper · explained in 5 episodes
• GRPO — 2× in this paper · explained in 6 episodes
• Generative AI — 2× in this paper · explained in 2 episodes
• Implicit Q-Learning — 2× in this paper
• Transformer — 2× in this paper · explained in 2 episodes
• AI Agents — 1× in this paper · explained in 2 episodes
• Agentic System — 1× in this paper · explained in 2 episodes
• Continual Learning — 1× in this paper · explained in 2 episodes
• EMMA — 1× in this paper · explained in 2 episodes
• Hierarchical Framework — 1× in this paper · explained in 2 episodes
• Markov Decision Process (MDP) — 1× in this paper · explained in 11 episodes
• Regularization — 1× in this paper · explained in 3 episodes

Sources

• Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs
• Broaden your SCOPE! Efficient Multi-turn Conversation Planning for LLMs with Semantic Space
• Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHF
• Reinforcement Learning from User Feedback
• Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations
• Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning
• Rewarding Chatbots for Real-World Engagement with Millions of Users
• Human-centric Dialog Training via Offline Reinforcement Learning
• Efficient RL for optimizing conversation level outcomes with an LLM-based tutor
• Proximal Policy Optimization Algorithms
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
• HybridFlow: A Flexible and Efficient RLHF Framework
• Direct Multi-Turn Preference Optimization for Language Agents
• Offline RL for Natural Language Generation with Implicit Language Q Learning
• CHAI: A CHatbot AI for Task-Oriented Dialogue with Offline Reinforcement Learning
• Reinforcing Language Agents via Policy Optimization with Action Decomposition
• Building Math Agents with Multi-Turn Iterative Preference Learning
• ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL
• Fine-Tuning Language Models from Human Preferences

Related papers

• Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
• AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
• Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
• Convergence issues in Relational Concept Analysis based on AOC-posets
• Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
• Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks