Beyond Policy Rollouts: Embedding-Space Directional Exploration with Anchored Noise for LLM RLVR
summary
The gist
The gist Beyond Policy Rollouts: Embedding-Space Directional Exploration with Anchored Noise for LLM RLVR proposes EDEN, an exploration method that intervenes at the rollout source through anchored
In short
EDEN is an exploration method for Large Language Model Reinforcement Learning from Human Feedback (RLVR) that intervenes at the input embedding stage to explore beyond a policy's standard rollouts. It works by conditioning the policy on perturbed inputs, combining Gaussian noise with an anchor embedding, and incorporating these informative trajectories into the policy updates.
Key concepts
- EDEN
- EDEN is an exploration technique that modifies how a language model learns by introducing controlled perturbations to the input embeddings before generation begins. Instead of relying only on what the policy generates itself, EDEN forces the model to explore different response directions by conditioning it on slightly altered inputs.
- Input Perturbations
- This involves changing the prompt's embedding at a single position immediately after it is processed but before any tokens are generated. This perturbation combines random Gaussian noise with a fixed anchor embedding shared across all perturbed samples, allowing the model to see how small input changes affect the final output trajectory.
- Absorb
- The Absorb component integrates these newly collected, informative trajectories into the training process of the main policy. It uses advantage-based selection and scaling to prioritize learning from these perturbed samples, ensuring that high-advantage, novel behaviors are effectively incorporated into the policy updates.
Terminology used across episodes
This episode discusses
- Beyond Policy Rollouts: Embedding-Space Directional Exploration with Anchored Noise for LLM RLVR · Paper Radio
- Learning to Explore with Parameter-Space Noise: A Deep Dive into Parameter-Space Noise for Reinforcement Learning with Verifiable Rewards
- Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models
- Training Verifiers to Solve Math Word Problems
- The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
- Measuring Mathematical Problem Solving With the MATH Dataset
- VADE: Variance-Aware Dynamic Sampling via Online Sample-Level Difficulty Estimation for Multimodal RL
- QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMs
- Diversifying RLVR Rollouts via First-Token Exploration · Paper Radio
- CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models
- ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
- Proximal Policy Optimization Algorithms
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay
- A Relative-Budget Theory for Reinforcement Learning with Verifiable Rewards in Large Language Model Reasoning
- Qwen2.5 Technical Report
- Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement
- Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration
- Diversity-Aware Policy Optimization for Large Language Model Reasoning
- DAPO: An Open-Source LLM Reinforcement Learning System at Scale
The paper
Beyond Policy Rollouts: Embedding-Space Directional Exploration with Anchored Noise for LLM RLVR · Read on arXiv
Daijiao Liu, Huiyi Wang, Lina Yao, Dong GongB
University of New South Wales (UNSW Sydney)
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: Today's paper: "Beyond Policy Rollouts".
Jane: The gist Beyond Policy Rollouts: Embedding-Space Directional Exploration with Anchored Noise for LLM RLVR proposes EDEN,
Tom: First, who's behind it and why it matters.
Paper summary: Tom: So to wrap up, this paper, “Beyond Policy Rollouts: Embedding-Space Directional Exploration with Anchored Noise for LLM RLVR,” basically shows that input-side perturbations beyond output-side exploration controls are effective in reinforcement learning with verifiable rewards.
Jane: It moves the focus of where you look for data from just sampling more rollouts to actively injecting specific, structured variations into the input embedding itself, which helps capture better solutions within the same rollout budget.
Lu: The authors show that this structured perturbation enables exploration without changing the model architecture, the rollout budget, or the test-time inference path <ref:2610.16105#pg2>.
Meng: It’s a way to get better performance metrics, like that eighty-three point two five Pass@four figure they mentioned, by being smarter about what input conditions you feed into the policy during training <ref:2610.16105#pg3>.
Tom: They prove that this approach works by showing gains across different model scales and reasoning benchmarks, suggesting it’s a viable technique for improving RLVR performance <ref:2610.16105#pg3>.
Jane: It suggests that the locus of exploration really matters; changing the conditions under which trajectories are generated can alter the effective trajectory distribution beyond simply drawing more rollouts from the same distribution <ref:2610.16105#pg2>.
Lu: It’s a targeted way to improve solution coverage, which is a central challenge in RLVR, by using these anchored noise techniques <ref:2610.16105#pg3>.
Meng: For deploying models, this means we could potentially use this input perturbation idea to find better behaviors without needing massive increases in our data collection pipeline <ref:2610.16105#pg3>.
Tom: That’s the gist of EDEN, using anchored noise at the embedding position immediately after the prompt to influence the whole rollout from the start <ref:2610.16105#pg1>.
Conclusion: Tom: So, we’re wrapping up this look at EDEN, which is called "Beyond Policy Rollouts: Embedding-Space Directional Exploration with Anchored Noise for LLM RLVR."
Jane: It really boils down to taking exploration beyond just sampling more trajectories from the policy itself.
Lu: The core idea is using input perturbations at the embedding position right after the prompt, before any tokens are generated.
Meng: So instead of just letting the model wander randomly during generation, you’re intentionally nudging it with some noise anchored to a shared point across all those perturbed samples.
Tom: Right, so it’s about injecting that controlled deviation into the input space immediately to influence the entire rollout from the very beginning.
Jane: It seems they've built a system where this input-side exploration works alongside standard output-side controls for reinforcement learning with verifiable rewards.
Lu: The method involves three main parts: first, perturbing samples under different conditions; second, controlling that noise with anchors and temperature schedules; and third, incorporating those informative perturbed trajectories back into the policy updates using advantage scaling.
Meng: From an engineering side, I’m interested in how they handle that selection process—how they filter those perturbed rollouts to make sure we only use the ones that actually give us a good signal.
Tom: Yeah, they use an advantage-gated filter and noise-aware advantage reweighting to select the best perturbed trajectories for training the policy.
Jane: The results show EDEN outperforms GRPO across different model sizes and reasoning benchmarks, with big gains in sampled solution coverage.
Lu: Specifically, on Qwen2 point five-Math-7B, it improved the average Pass@four from eighty-one point sixty-five to eighty-three point twenty-five when the temperature was set at one.
Meng: That improvement in Pass@four is solid, but what about deployment? Does this need us to collect way more data just to get those better rollouts?
Tom: No, it’s structured perturbation, so it doesn't change the model architecture or how we run inference at test time. It’s an exploration technique within the training loop.
Jane: It suggests that for RLVR, we can achieve better solution coverage without significantly increasing our rollout budget or changing our deployment pipeline.
Lu: The authors conclude that these input-side perturbations provide effective gains in RLVR by offering a structured way to explore beyond standard output-side controls.
Meng: So the implication is that this isn't just about sampling more; it’s about intelligently guiding the model toward better solutions through tailored input variations.
Tom: Exactly. We should keep watching how this specific type of input anchoring translates when we try to apply it to other complex reasoning tasks.
More episodes
- 2610.13750-Studying the Effects of Task Diversity Using Small Mathematical Worlds
- 2610.13787-Optimal Algorithms for Nonsmooth Nonconvex Stochastic Optimization with Constraints
- 2610.13799-Adaptive Variance Reduction without Knowing Oracle Smoothness
- 2610.13834-On the Design Space of Continuous Diffusion Language Models
- 2610.13956-MDN-VJEPA: Multimodal Probabilistic Joint-Embedding World Models
- 2610.14008-Generalized PINN-Based Inverse Estimation of Spatially Varying Heat Transfer Coefficients with Transfer Learning
- 2610.14016-DShuttleNet: A Badminton Doubles Dataset for Evaluating Team Coordination
- 2610.14024-Beyond a Better Score: Long-Horizon Agentic ML Development and Evaluation Protocol for Physics Time Series
- 2610.14098-From Prototype to Massing: Learning Graph Correspondence for Scalable Facade Modeling
- 2610.14120-Early Detection of Pension-Sponsor Distress from Public Filings: A Sponsor Vulnerability Index, Stress-Augmented VaR, and Dynamic Self-Investment Caps