GRPO-QPS: Target-Preserving Reinforcement Learning for Quantum Posterior Sampling
cs.LG, quant-ph
Submitted: 2026-09-13
Updated: 2026-09-21
License: http://creativecommons.org/licenses/by/4.0/
The gist: Reward-based learning can alter the very posterior distribution that scientific inference aims to estimate.
Terminology
Abstract
Reward-based learning can alter the very posterior distribution that scientific inference aims to estimate. GRPO-QM sidesteps this by learning only an exploration strategy for a stated quantum-tomography posterior: a group-relative policy chooses among reversible physical moves, and an exact Metropolis correction ensures the posterior remains stationary once the policy is fixed. We then examine what learning contributes beyond physical proposal mechanisms and prior knowledge. Reconstruction comparisons suggest that most of the gains over the tested flows come from those two components rather than from learning itself, and a closed-form counterexample explains why: a reward tied to accepted motion can increase even when a physical observable remains highly correlated. Reward comparisons that control for initialization also show that aggregating across posteriors can invert the ranking that a reward implies within any single posterior. Lastly, on 45 enumerated posteriors with three training seeds each, exact and sampled gradients of the same trajectory objective reduce mean physical estimation variance by 12.63% and 6.32% relative to a tuned mixture, while averaging the trajectory score without rescaling its penalty yields only 0.43%. This indicates that objective scaling accounts for part of the disparity between sampled and exact training. The regained benefit is concentrated at four shots and flips sign at sixteen, so these exact, visible-bank diagnostics identify a concrete, reproducible failure mode in sampled training, the objective-scaling fix that addresses part of it, and the remaining gap.
Sources
- Markovian Flow Matching: Accelerating MCMC with Continuous Normalizing Flows
- Flow-GRPO: Training Flow Matching Models via Online RL
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Harnessing the Power of Reinforcement Learning for Adaptive MCMC
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks