MGE-PO: Entropy-Aware Credit Assignment for Mathematical Reasoning

arXiv:2610.23245 · cs.CL, cs.AI · Submitted 2026-10-16 · Read on arXiv

cs.CL, cs.AI

Submitted: 2026-10-16

Updated: 2026-10-16

License: http://creativecommons.org/licenses/by/4.0/

The gist: GRPO assigns the same group-relative advantage to equally rewarded responses within a question, leaving their generation uncertainty unused.

Terminology

Abstract

GRPO assigns the same group-relative advantage to equally rewarded responses within a question, leaving their generation uncertainty unused. We introduce MGE-PO, centered on four-quadrant (4Q) advantage shaping that combines verified correctness with rollout generation entropy. Correctness determines the sign of an additional correction, while entropy controls its magnitude: lower-entropy successes receive larger bonuses and higher-entropy failures receive larger penalties. An online entropy-based question curriculum supports this credit rule using cached rollout measurements. At the final checkpoint after eight training epochs on Qwen3-8B-Base, adding 4Q across three policy optimization settings yields absolute gains of 3.23-3.69% in six-benchmark macro avg@32 and 4.24-11.06% in macro pass@32. On Qwen2.5-7B-Base, the complete configuration raises four-benchmark macro avg@32 from 41.5% to 45.4%.

Related papers