GMTS: Gradient Magnitude-based Token Selection Improves RLVR Training for LLM Reasoning

arXiv:2608.30632 · cs.CL, cs.AI, cs.LG · Submitted 2026-08-31 · Read on arXiv

cs.CL, cs.AI, cs.LG

Submitted: 2026-08-31

Updated: 2026-08-31

Code: https://github.com/outongyiLv/GMTS

Terminology

• GRPO — 36× in this paper · explained in 6 episodes
• Benchmark — 13× in this paper · explained in 2 episodes
• Accuracy — 12× in this paper · explained in 1 episode
• Reinforcement Learning — 7× in this paper · explained in 4 episodes
• Gradient Magnitude-based Token Selection — 3× in this paper
• Regularization — 3× in this paper · explained in 3 episodes
• Computational overhead — 2× in this paper · explained in 2 episodes
• KL Regularization — 2× in this paper · explained in 2 episodes
• Objective Function — 2× in this paper · explained in 2 episodes
• Reinforcement Learning (RL) — 2× in this paper · explained in 25 episodes
• Ablation — 1× in this paper · explained in 1 episode
• Ablation study — 1× in this paper · explained in 1 episode
• Adaptability — 1× in this paper · explained in 1 episode
• Consistency — 1× in this paper · explained in 3 episodes
• Entropy Collapse — 1× in this paper · explained in 2 episodes
• Fine-Tuning — 1× in this paper · explained in 5 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• Group Relative Policy Optimization — 1× in this paper
• Group Relative Policy Optimization (GRPO) — 1× in this paper · explained in 5 episodes
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
• Reward Shaping — 1× in this paper · explained in 2 episodes
• Score Function — 1× in this paper · explained in 4 episodes
• Verifiable Rewards — 1× in this paper · explained in 2 episodes

Sources

• GPT-4 Technical Report
• LLaDA2.1: Speeding Up Text Diffusion via Token Editing
• SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization
• The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
• CURE: Critical-Token-Guided Re-Concatenation for Entropy-Collapse Prevention
• GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization
• Which Tokens Matter? Adaptive Token Selection for RLVR with the Relative Surprisal Index
• FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimization
• Proximal Policy Optimization Algorithms
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
• VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model
• CommonsenseQA 2.0: Exposing the Limits of AI through Gamification
• GTPO and GRPO-S: Token and Sequence-Level Reward Shaping with Policy Entropy
• Emergent Hierarchical Reasoning in LLMs through Reinforcement Learning
• Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVR
• Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning
• VisualQuality-R1: Reasoning-Induced Image Quality Assessment via Reinforcement Learning to Rank
• Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs
• Diversity-Aware Policy Optimization for Large Language Model Reasoning
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering