GCPO: Diagnosing and Constraining Subspace Geometry in Rollout RL for LLMs

GCPO: Diagnosing and Constraining Subspace Geometry in Rollout RL for LLMs

Abstract

On-policy rollout methods such as GRPO are central to posttraining of large language models. Yet, they frequently suffer from training instabilities, cross-task capability degradation, and response-length inflation. Although prior work has characterized the subspace geometry of aggregate updates, the stepwise variation of this geometry and its relationship to model performance remain unclear. We introduce Principal-Subspace Overlap, a dimension-corrected measure of individual rollout updates relative to the dominant singular subspaces of pretrained weights. Despite low average overlap, transient spikes often precede performance degradation. To address this, we propose GCPO (Geometrically Constrained Policy Optimization), which applies hard bilateral orthogonal projections to constrain updates to the complementary subspaces, preventing such excursions by construction. Across mathematical reasoning, code generation, and tool-use tasks on Qwen3-8B and GLM4-9B, GCPO consistently outperforms GRPO and recent variants, including DAPO and GSPO, improving over the base models and the strongest baseline by up to 27.69 and 2.37 points, respectively. Furthermore, GCPO preserves general capabilities, eliminates response-length inflation, and stabilizes policy entropy. Our findings provide a new diagnostic lens and a principled design perspective for stable reinforcement learning post-training.

Introduction

Rollout-based reinforcement learning (RL) has become a widely used post-training mechanism for improving large language models (LLMs) on mathematical reasoning, coding, and tool use. Its optimization data, however, are generated by the evolving policy itself: each update changes both the policy and the rollout distribution used to construct the next update. This feedback loop can yield unstable training, degradation on capabilities outside the optimized task, and response-length inflation.

Most existing remedies stabilize RL through objective- or policy-level controls. KL regularization constrains divergence from a reference policy, clipping limits large likelihood-ratio changes, and reward design modifies the optimization signal. While effective, these methods may treat different updates as similar if they have similar policy-level statistics, even when their directions in parameter space differ. Because updates of similar magnitude may interact differently with the structured parameter space of the pretrained model, we ask whether their stepwise directions provide a complementary diagnostic of training dynamics.

We focus on the dominant singular subspaces of each pretrained linear weight matrix. The top right and left singular vectors identify the dominant input and output directions, respectively. Together, they define the optimal rank-k approximation of the pretrained transformation. Updates overlapping these subspaces can therefore alter structurally prominent components of the pretrained transformation. Rather than interpreting these directions as a literal decomposition of semantic knowledge, we treat them as a functionally distinguished yet tractable structural reference. Prior analyses further show that RL updates are predominantly off-principal when averaged over training. This aggregate regularity leaves open an important question: do individual updates exhibit transient principal-subspace overlap, and how is such overlap related to training performance?

Our stepwise analysis reveals transient entry of individual updates into the pretrained principal subspaces. We decompose each realized update into four blocks according to its left- and right-side overlap with these subspaces, and subtract the overlap expected under an isotropic null. Although most update energy lies in the doubly orthogonal block, aggregate statistics obscure intermittent spikes in excess principal-subspace overlap. Across the runs we examine, sustained or repeated spikes coincide with, and often precede, declines in validation accuracy. We therefore treat overlap as a geometric correlate of unstable phases, motivating updates that preserve the selected pretrained mappings by remaining in their bilateral orthogonal complement.

We instantiate this hypothesis in GCPO (Geometrically Constrained Policy Optimization), which augments rollout-based policy optimization with a hard geometric constraint on the effective policy update. At each rollout iteration, GCPO optimizes the underlying policy objective while requiring every adapted layer update to lie in the bilateral orthogonal complement of the selected pretrained principal subspaces. GCPO thus changes the admissible directions of policy improvement, rather than introducing another policy-level penalty. The constraint is complementary to KL regularization: KL controls policy change in output space, whereas GCPO controls the feasible directions of parameter change. It also has a precise layer-level guarantee: for inputs in a selected principal input subspace, the adapted layer's response is unchanged. The remaining complement is large, retaining a high-dimensional feasible complement for task adaptation.

Comprehensive evaluations on Qwen3-8B and GLM4-9B cover mathematical reasoning, code generation, and tool use. Across all tasks, GCPO achieves the best accuracy, outperforming the strongest baseline by 1.02–2.37 points (±std over 3 seeds) and the corresponding instruction-tuned model by 7.09–27.69 points. When trained on mathematics and evaluated on other tasks, it attains the strongest worst-case retention. It also shows smoother accuracy and policy-entropy trajectories and suppresses response-length inflation.

In summary, the main contributions are threefold:

• We introduce a dimension-corrected, stepwise measure of principal-subspace overlap and show that elevated overlap is a warning signal associated with declining validation performance in rollout RL.

• We formulate rollout RL as a constrained policy optimization problem and propose GCPO to enforce bilateral orthogonality on every effective policy update, which preserves the principal pretrained mappings while retaining a large complementary update space.

• Across two model families and three task domains, GCPO outperforms all evaluated baselines, better preserves cross-task capabilities, and exhibits more stable accuracy, policy-entropy, and response-length dynamics.

Related Work

Rollout-based RL for LLM Alignment: Building upon PPO, recent advances like GRPO, GSPO, GMPO, and DAPO have significantly improved advantage estimation and scaling efficiency. Despite these innovations, the dynamic nature of self-generated rollouts frequently drives policies toward optimization instabilities, response-length inflation, and general capability degradation. Crucially, existing methods primarily intervene through objectives or observable policy behavior, leaving the geometry of realized parameter updates less explored.

Regularization and Stability in RL: RL post-training is commonly stabilized through objective- or policy-level controls. KL penalties discourage deviation from a reference policy, clipping constrains large likelihood-ratio changes, and reward shaping adjusts the scalar optimization signal to reduce undesirable behaviors such as reward hacking or length inflation. These techniques are effective, but they remain soft controls: they discourage unstable updates rather than ruling them out. Their effectiveness often depends on carefully tuned penalty coefficients and can be weakened by the high-variance feedback of rollout-based RL.

Geometric Analysis of Policy Optimization: Recent studies suggest that the geometry of parameter updates plays a critical role in RL-based post-training. Rather than treating policy updates as unstructured perturbations, these studies show that RL-induced updates exhibit systematic geometric patterns relative to the pretrained weights. In particular, recent analyses find that successful RL updates are, on average, more concentrated outside the dominant singular subspaces of the pretrained operators. This off-principal tendency suggests that effective policy adaptation may preferentially exploit directions that interfere less with the dominant structures learned during pre-training. However, existing analyses mainly characterize aggregate update behavior, leaving open how stepwise deviations from this geometry emerge during training and whether they are related to optimization instability.

Principal-Subspace Overlap and Instability

Prior work finds that RL updates, when aggregated over training, predominantly lie outside the dominant singular subspaces of pretrained weights. Nevertheless, aggregate statistics may obscure intermittent spikes, termed excess principal-subspace overlap, which often precede validation degradation. Controlled interventions further show that amplifying this overlap causes dose-dependent accuracy drops, motivating complementary-subspace constraints.

For a pretrained weight matrix Wref ∈ Rdout ×din, let Wref = ΦΣΨ⊤ be its SVD. The spans of the top-k left and right singular vectors, Φk and Ψk, are its principal subspaces. The realized update δ(t)W = Wt − Wt−1 is decomposed into four mutually orthogonal blocks: PP, PO, OP, and OO, where OO is orthogonal to the principal subspace on both sides. The overlap statistic is defined as Ot = (EPP + EPO + EOP)/Etotal = 1 − EOO/Etotal.

This raw ratio depends partly on dimensionality: an isotropic update has nonzero overlap simply because the principal subspaces have dimension k. For a dout × din matrix, its expected overlap is Onull = 1 − ((dout−k)(din−k))/(dout din). The dimension-corrected quantity is Otexcess = Ot − Onull. Positive Otexcess means that an update is more aligned with the principal subspaces than an isotropic update of the same size.

Figure 1 shows two consistent findings. First, most update energy remains in OO (93.8% for Qwen3-8B and 97.7% for GLM4-9B on average), agreeing with the aggregate off-principal trend. Second, this average masks intermittent positive spikes in Otexcess. In the displayed runs, sustained or repeated spikes coincide with, and precede, declining validation accuracy; the effect appears as gradual deterioration in one setting and a sharp drop in another. Layer-wise measurements further show that these spikes are concentrated rather than uniform, often in intermediate and upper layers.

To move beyond correlation, a controlled intervention was performed on a GRPO update of Qwen3-8B on ToolAlpaca. The principal-overlapping component was rescaled by η while preserving the Frobenius norm of every layer-wise update. Increasing overlap produces a clear dose-dependent accuracy drop. Relative to the original update (η = 1, 56.89%), removing the overlapping component improves accuracy, whereas principal-subspace injection substantially reduces accuracy. A matched random-subspace intervention causes substantially less degradation. These results provide local intervention evidence that increasing principal-subspace overlap can directly harm model performance, beyond the effects of update magnitude or arbitrary directional perturbation.

Methodology

The diagnostic suggests a simple intervention: rather than modifying the rollout objective, restrict each policy update to the bilateral orthogonal complement of the dominant singular subspaces of the pretrained weights. GCPO realizes this intervention within GRPO through a projected low-rank parameterization.

Let θ0 be the pretrained policy and M the set of adapted linear layers. For each l ∈ M, the rank-k singular subspaces Φk(l) and Ψk(l) of Wref(l) are computed once. The objective is to maximize the usual rollout objective while having zero principal-subspace overlap at every adapted layer:

max Jrollout(πθ0+∆θ) − βDKL(πθ0+∆θ πref), subject to Φk(l)⊤ δ(t)W(l) = 0 and δ(t)W(l) Ψk(l) = 0, ∀l ∈ M.

The two constraints are equivalent to retaining only the OO block of each update. They complement, rather than replace, KL regularization: KL controls policy change in output space, whereas the constraints control where the parameter update can lie. A penalty could encourage these constraints, but would require trading them against the rollout objective. GCPO instead enforces them exactly by parameterization.

For each adapted layer, with fixed orthogonal-complement projectors ΠΦ⊥, ΠΨ⊥, the update is parameterized as δ(t)W(l) = αΠΦ⊥(l) L(l) R(l) ΠΨ⊥(l), where L(l) and R(l) are trainable low-rank factors and α is the standard scaling constant. The right projector removes components acting on dominant input directions; the left projector removes components written into dominant output directions. This parameterization satisfies both constraints for every value of the trainable factors. Thus, even if rollout gradients contain principal-subspace components, those components cannot change the effective layer update. This is the central property of GCPO: it converts the observed preference for off-principal updates into an exact feasible set, while leaving optimization within that set unchanged.

GCPO satisfies the bilateral constraints at every adapted layer and every optimization step. Consequently, the pretrained layer mapping is unchanged for inputs in the selected principal input subspace, with an analogous guarantee for the principal output subspace. The feasible update space has dimension (dout−k)(din−k), and every feasible rank-r update can be represented by the projected low-rank parameterization. Thus, for k small relative to the layer width, GCPO retains a large space for task adaptation.

Although GCPO uses low-rank factorization for efficient implementation, it solves a different constrained optimization problem from conventional low-rank adaptation. LoRA primarily restricts update rank for parameter efficiency, while some geometric variants shape update directions to prevent catastrophic forgetting typically under fixed-data supervised fine-tuning. In contrast, GCPO is motivated by transient principal-subspace overlap under on-policy rollout feedback in RL, where each policy update changes the subsequent training distribution. It therefore constrains update direction through bilateral orthogonality to stabilize stepwise RL dynamics and preserve pretrained mappings, while low rank serves mainly as an efficient parameterization of this feasible space.

Experiments

Models and Tasks: Two widely used instruction-tuned LLMs are adopted as backbone models: Qwen3-8B and GLM4-9B, evaluated across three representative domains:

• Mathematical Reasoning: Evaluated on MATH500. The reward is computed via exact matching of the final boxed answer using rule-based parsing.

• Code Generation: Evaluated on HumanEval+ for Python function synthesis. The reward is granted based on the execution pass rate across unit tests.

• Tool Use: Evaluated on ToolAlpaca, which requires generating the appropriate API call given a tool specification and a user request. The reward is based on exact function-name matching, argument-key consistency, and normalized argument-value matching.

Baselines: GRPO, objective-level variants including GSPO, DAPO, and GMPO, and GRPO-LoRA (unconstrained low-rank adaptation without the hard bilateral orthogonality enforced by GCPO). GRPO-LoRA and GCPO use the same adaptation rank and scaling configuration.

Implementation Details: For all GRPO-based methods, K = 16 rollouts per prompt are sampled during training. For GCPO, the top-k = 8 singular subspaces are precomputed and updates are constrained to their bilateral orthogonal complements. Bootstrap-estimated majority@16 accuracy from 16 test-time responses per example is reported.

Main Results

Table 1 reports the mean and standard deviation over three independent training seeds across three task domains and two backbone models. GCPO achieves the best accuracy in all six model–task settings. Relative to the corresponding instruction-tuned base models, it improves accuracy by 7.09–27.69 percentage points, and surpasses the strongest competing method on each benchmark by 1.02–2.37 points. Moreover, GCPO exhibits the lowest standard deviation in all six settings, indicating reduced sensitivity to training stochasticity. The margins are particularly pronounced on GLM4-9B, where GCPO outperforms the strongest baselines by 2.15–2.37 points.

On Qwen3-8B, GCPO obtains an average accuracy of 78.63, exceeding the strongest baseline average of 77.48 by 1.15 points. On GLM4-9B, it improves the strongest baseline average from 73.73 to 76.12. Averaged across all six model–task settings, GCPO reaches 77.38, compared with 75.61 for the strongest baseline. GRPO-LoRA with the same rank achieves consistently lower accuracy than GCPO across all evaluations, confirming that the explicit directional orthogonal projection is the primary driver of the method's superiority.

Cross-Task Capability Preservation

Single-domain RL post-training may improve the target task while degrading other capabilities. Models are post-trained on MATH500 and evaluated on code generation (HumanEval+) and tool-use (ToolAlpaca) benchmarks.

Existing RL variants poorly retain tool-use capability. GRPO and its variants generally exhibit substantial degradation on ToolAlpaca after MATH500 post-training. This drop is most severe for GRPO, with a large decline of −14.97 on GLM4-9B. Mathematical reasoning can transfer to code generation; RL training on math tasks has a milder effect on coding capabilities, with several variants even improving code generation performance.

GCPO best preserves general capabilities. GCPO obtains the best Worst ∆ scores on both Qwen3-8B and GLM4-9B, with gains of +1.03 and +0.91, while also improving HumanEval+ by +3.99 and +5.88, respectively. These results indicate that constraining updates away from dominant pretrained subspaces better balances task-specific RL adaptation with the preservation of general capabilities.

Optimization Dynamics of GCPO

Robust Training Stability: GRPO exhibits severe performance oscillations and high-variance fluctuations on ToolAlpaca for Qwen3-8B, whereas GCPO yields a smooth, continuously increasing accuracy curve. The bilateral orthogonal constraint effectively acts as a structural barrier, preventing variance-heavy gradients from intruding into the principal subspace.

Stable Policy Entropy Dynamics: GRPO exhibits large entropy oscillations on MATH500 with GLM4-9B, while most baselines exhibit rapid entropy decay, suggesting premature overconfident exploitation. GCPO instead maintains a smooth, gradual decay, suggesting more controlled policy specialization without rapid loss of diversity.

Resistance to Response-Length Inflation: GRPO suffers from aggressive and uncontrolled output length expansion on MATH500. Conversely, GCPO substantially reduces response length, maintaining concise and stable generation lengths. Shielding dominant pretrained directions—including those encoding length priors—from high-variance gradients reduces the policy's tendency to exploit verbosity as a reward-hacking shortcut.

Superior Memory Efficiency: Full-parameter GRPO requires substantially more memory because optimizer states and gradients are maintained for all adapted dense weights. Both GRPO-LoRA and GCPO reduce this cost through low-rank adaptation. GCPO achieves comparable peak memory to LoRA, showing that the bilateral geometric constraint does not sacrifice the memory efficiency of parameter-efficient training.

Ablation Studies

Orthogonal Constraints Prevent Capability Collapse: Forcing updates into the Principal subspace triggers a severe performance collapse, indicating that altering dominant singular directions disrupts task-relevant capabilities. Conversely, the Orthogonal complement achieves peak accuracy, substantially outperforming Random projections.

Bilateral Projections Seal Collapse Pathways: Unilateral constraints on solely the left (output) or right (input) singular spaces yield only marginal gains over the unconstrained baseline. The bilateral projection is mathematically necessary to seal both pathways simultaneously, ensuring zero principal-subspace overlap and optimal accuracy.

Hard Constraints Survive Iterative Rollout Dynamics: Both KL divergence and explicit soft orthogonality penalties underperform the hard constraint approach. Under the iterative feedback loops of RL, dominant task gradients inevitably overwhelm soft loss terms, allowing overlap to persist. By enforcing orthogonality by construction, the hard constraint robustly shields the pretrained manifold.

The Protected Dimension Balances Protection and Adaptation: Performance varies substantially with k and peaks at k = 8 on both MATH500 and ToolAlpaca. Smaller k provides insufficient protection, whereas larger k overly restricts the feasible update space.

Limitations

The study focuses on the parameter-space dynamics of on-policy RL, and it remains unclear whether the observed geometric patterns also extend to other post-training paradigms, such as DPO, KTO, and OPD. Although principal-subspace overlap is consistently associated with performance degradation, its causal relationship with model instability, rollout behaviors such as length inflation and reward hacking is not yet fully understood. Moreover, future work should examine broader alignment objectives, model scales, and training settings, while adaptive, layer-wise selection of k, to assess the generality of this principal-subspace overlap.

Conclusion

The paper studies rollout-based RL instability through the geometry of step-wise parameter updates. Transient principal-subspace overlap is identified as a warning signal associated with performance degradation, with controlled interventions providing local evidence of its harmful effect. Based on this diagnosis, GCPO is proposed, which restricts effective updates to the bilateral orthogonal complements of pretrained principal subspaces. Across two backbones and three task domains, GCPO achieves the highest mean performance overall, while improving capability retention, response-length control, and policy-entropy stability. These results highlight parameter-space geometry as a useful diagnostic and design perspective for stable LLM reinforcement learning.

Based on this paper, here are specific improvements you can make to AI systems:

• Implementation: Modify existing RL post-training frameworks (GRPO, PPO, DAPO) to project all parameter updates onto the bilateral orthogonal complement of the top-8 singular subspaces of each pretrained weight matrix.

• Resulting Capability: The AI system will maintain stable training dynamics, avoid performance collapses, and prevent capability degradation on non-target tasks during RL fine-tuning.

• Implementation: Add a diagnostic module that computes the dimension-corrected excess overlap statistic (O excess) after each training step, with alerts triggered when sustained positive spikes occur.

• Resulting Capability: The system can detect early warning signs of impending performance degradation before validation accuracy drops, allowing for proactive intervention (e.g., reducing learning rate, reverting to checkpoint).

• Implementation: Replace KL penalties and soft orthogonality losses with hard constraints enforced via parameterization (projected low-rank factorization) rather than loss terms.

• Resulting Capability: The AI system becomes robust to gradient overwhelm in iterative rollout feedback loops, eliminating the need for carefully tuned penalty coefficients that can fail under high-variance RL gradients.

• Implementation: Apply the bilateral orthogonal projection specifically to layers that encode length priors and verbosity patterns, preventing high-variance gradients from exploiting these directions.

• Resulting Capability: The system maintains concise, stable response lengths without explicit length penalties, reducing reward-hacking behaviors while preserving generation quality.

• Implementation: When fine-tuning on a single domain (e.g., math), constrain updates to avoid dominant singular directions of pretrained weights to maintain tool-use and code-generation capabilities.

• Resulting Capability: The AI system can be specialized on one task without catastrophic forgetting of other capabilities, achieving positive transfer (+1.03 to +5.88 points on non-target tasks).

• Implementation: Use the geometric constraint as a structural barrier to prevent premature entropy collapse during RL training.

• Resulting Capability: The system maintains controlled exploration throughout training, avoiding overconfident exploitation and preserving output diversity until convergence.

• Implementation: Combine the bilateral orthogonal projection with low-rank factorization (rank 8) to achieve full-parameter stability with LoRA-level memory efficiency.

• Resulting Capability: The system can perform stable RL post-training on large models with reduced GPU memory requirements, making it feasible for resource-constrained environments.

• Implementation: Package the bilateral orthogonal projection as a reusable module that can be plugged into any parameter-update scheme (not just RL) to prevent updates from altering dominant pretrained transformations.

• Resulting Capability: Any fine-tuning or adaptation process becomes inherently safer, reducing the risk of catastrophic forgetting or capability degradation across diverse downstream tasks.