GRPO for Financial Advice Generation: Outperforming Commercial LLMs under CATE Evaluation

Generating actionable financial advice from business records demands that models integrate numerical reasoning, domain knowledge, and sound judgment, while avoiding recommendations that could harm the business. Direct supervision is difficult: historical decisions are not necessarily optimal, and high-quality free-form labels are expensive to obtain. We formulate financial advice generation as a reinforcement learning problem and fine-tune an open-weight language model using Group Relative Policy Optimization (GRPO). Our reward is an LLM-as-a-judge rubric that scores each recommendation across multiple binary dimensions of advice quality, augmented with a safety gate for harm prevention.

Since LLM-based evaluation alone cannot confirm whether improvements reflect genuine business value rather than adaptation to the judge, we complement it with a judge-independent audit based on a standard doubly-robust Conditional Average Treatment Effect (CATE) estimator. Under this observational off-policy audit, our trained LLM achieves approximately twice the estimated gross-profit lift of the strongest evaluated commercial baseline (0.0228 vs. 0.0104), together with the lowest downside rate and the least negative tail risk of any policy evaluated. Notably, the two evaluations do not rank the baselines identically: the untrained base model places last on the judge rubric but second on the causal audit, indicating that the audit captures a signal the judge does not. Our results demonstrate that GRPO with a finance-grounded reward signal can produce substantially more useful business recommendations than commercial LLMs, and that a judge-independent causal audit is a valuable complement to, rather than a confirmation of, LLM-as-a-judge assessment in financial NLP.

In this work, our main contributions are: (1) rubric-grounded, safety-gated GRPO for financial advice: we formulate free-form business financial advice generation as an RL problem and fine-tune an open-weight LLM with a finance-specific, safety-gated rubric reward; (2) a natural-language-to-action causal audit: we map generated advice into a fixed business-action taxonomy and apply a standard doubly-robust CATE estimator to audit whether recommendations correspond to historically beneficial actions; and (3) a dual evaluation against commercial and open-weight baselines, using both judge-based rubric scores and a judge-independent score from a standard doubly-robust CATE estimator, with uncertainty, downside-rate, and tail-risk metrics; we show that the two evaluations agree on our policy but diverge on the baselines, and argue that this divergence is itself evidence that the audit is not a proxy for the judge.

We treat advice generation as a reinforcement learning problem and learn a policy πθ (a s, g) that maps a state and goal to a recommendation. This framing is motivated by two properties of the domain. First, the historical actions logged for each business are not optimal targets, as they reflect the constraints and incomplete information available at decision time, so supervised imitation would propagate suboptimal behavior. Second, recommendations are open-ended natural language, ruling out value-based methods over a fixed action set. RL instead lets us optimize a scalar reward that directly encodes what makes a recommendation useful.

We optimize πθ with Group Relative Policy Optimization (Shao et al., 2024). For each prompt (s, g) we sample a group of K candidate recommendations a1,..., aK from the current policy, score each with the reward function of Section 3.3 to obtain rewards R1,..., RK, and compute a group-relative advantage by standardizing rewards within the group, Ai = (Ri − mean(R1,..., RK)) / std(R1,..., RK). The policy is then updated with the clipped GRPO objective, which avoids training a separate value model. We additionally anchor the policy to the base model with a KL penalty weighted by β, which we found necessary to prevent policy drift during training.

The reward is provided by an LLM judge (Claude Opus 4.5) that scores a recommendation against a rubric of eleven binary criteria: a safety gate plus ten quality criteria organized into six aspects. The safety gate is critical: if implementing the recommendation would cause significant harm to the business (e.g., eliminating a primary revenue stream or creating unsustainable cash-flow risk), the gate hard-zeros the reward. Two penalties shape the final reward. Recommendations whose JSON cannot be parsed receive a fixed penalty of −0.5. In addition, to discourage degenerate over-long reasoning, a thinking-length penalty p(a) ∈ [0, 0.2] is applied that is zero below a soft cap on reasoning length and ramps linearly to its maximum at a hard cap. The final reward is R(a) = c(a) − p(a).

The training reward comes from an LLM judge. As a result, a higher judge score may reflect recommendations that appeal to the judge rather than genuinely better business outcomes. To test for this, we audit the learned policy with a judge-independent estimate of its effect on real outcomes. This estimate is computed entirely from logged observational data. The key step is to bridge from free-text recommendations to a quantity for which causal effects can be estimated. An independent action mapper assigns each recommendation to a discrete business action; the resulting action is treated as the treatment, the business-state embedding as the conditioning covariates, and the realized year-over-year gross-profit growth as the outcome. We then estimate the conditional average treatment effect (CATE) of the policy’s actions using a standard doubly-robust (AIPW) estimator (Robins et al., 1994; Bang and Robins, 2005; Dudík et al., 2011; Abrevaya et al., 2015), and report the estimated lift alongside a downside rate and a tail-risk measure. We use these estimates as a judge-independent policy-ranking audit rather than as evidence of realized, deployed business impact.

We train on logged business financial states paired with the goal of improving gross profit. The base policy is Qwen3.5-27B, fine-tuned with LoRA under DeepSpeed ZeRO-2. Personally identifying entities are removed from the training data and replaced with synthetic surrogates. GRPO uses K=12 candidate generations per prompt, a maximum completion length of 8,000 tokens, a KL coefficient β=0.001, learning rate 5×10−5 with a cosine schedule, and the doubly-robust GRPO loss variant. The judge reward model is Claude Opus 4.5. We select the best checkpoint on a held-out validation reward.

We use two complementary evaluations. The first is the LLM-as-a-judge rubric score, reported on a held-out set of 500 businesses with 5 independent trials per business (n=2,500 per model); we report the mean over per-trial means with a 95% confidence interval. The second is the causal off-policy evaluation, which we instantiate below. We compare against commercial LLMs (Claude Opus 4.5/4.6, Claude Sonnet 4.5, GPT-5.4) and the untrained Qwen3.5-27B base model.

The audit proceeds in two stages. In stage one, on logged data where the treatment Ai and the outcome Yi are both observed, we evaluate the AIPW pseudo-outcome for every logged business and average it within each (action, propensity-decile) stratum. Strata that fail minimum-support or effective-sample-size gates are marked invalid and excluded. This yields a table of stratum-level effect estimates spanning the 49 actions that clear the eligibility threshold and 10 propensity bins per action. In stage two, each held-out recommendation is mapped to an action, the propensity of its business state for that action is predicted and assigned to a bin, and the recommendation receives the corresponding stratum-level effect. The estimand is thus the mean stratum-level effect of the (action, propensity-bin) cells into which a policy’s recommendations fall, and policies differ only through which actions they select and how their states distribute across propensity bins.

The three reported metrics are computed over this set of per-recommendation values. The estimated lift (DR-CATE) is their mean. The downside rate (DR%) is the fraction whose assigned stratum effect is negative. The conditional value-at-risk CVaR0.10 is the mean over the worst 10%. Higher lift is better; a lower downside rate and a less negative CVaR0.10 are better. The audit follows the same protocol as the judge evaluation. Each policy is evaluated on 500 randomly sampled held-out business states over 5 independent runs with different random seeds. We compute the three metrics within each run, and report the mean across runs together with a 95% confidence interval over the five run-level estimates. The mapper, the propensity model, and the outcome model are held fixed across all policies, so every system is scored by the same instruments.

Recommendations are mapped to the action taxonomy by a separate LLM classifier, distinct from and independent of the Claude Opus 4.5 reward judge, so that mapping noise cannot directly inflate the audit. Given a recommendation and the fixed action catalogue, the mapper returns the single best-matching action together with a confidence score; recommendations that fall below a confidence threshold, or that match no catalogue action, are assigned a dedicated no-match label and excluded from scoring rather than forced onto an ill-fitting action.

For the judge evaluation, after GRPO training, Qwen3.5-27B reaches 9.514, the top score on the leaderboard, ahead of Claude Opus 4.6 (9.365), GPT-5.4 (8.949), and Claude Sonnet 4.5 (8.712), and well above the untrained base model (8.457). Two caveats apply to this table. First, the judge belongs to the Claude family, which may introduce a small self-preference bias in favor of Claude baselines; this makes the comparison conservative for our model. Second, and more importantly, our policy was trained against this same judge, so its rubric score is in part a measure of successful optimization rather than an independent assessment. We therefore read Table 3 as evidence that training achieved its objective, and treat the judge-independent causal audit below as the primary comparative evidence.

For the causal evaluation, our policy attains an estimated gross-profit lift of 0.0228 [0.0211, 0.0246], approximately twice that of the strongest commercial baseline, Claude Opus 4.6 (0.0104, a ratio of 2.20×), together with the lowest downside rate (0.155 vs. 0.232) and the least negative tail risk (CVaR0.10 = −0.073 vs. −0.100), meaning that even its worst-decile recommendations are less harmful on average. Our policy ranks first on all three metrics, and its confidence interval does not overlap that of any other policy on any metric; under a Welch t-test over the five per-trial means, every pairwise comparison with our policy is significant at p < 0.01.

The commercial baselines separate clearly under the audit. Claude Opus 4.6 is the only commercial system with a lift that is both positive and comfortably distinguishable from zero (p < 0.001); GPT-5.4 is positive but noisier (0.0082, p = 0.015, and the widest interval of any policy); Claude Sonnet 4.5 is not distinguishable from zero (0.0028 [0.0000, 0.0055], p = 0.051); and Claude Opus 4.5 is estimated to be negative (−0.0025 [−0.0043, −0.0007]), i.e. its recommendations map to actions that are, on average, associated with slightly worse subsequent gross-profit growth. Downside rate tracks this ordering closely: the three weakest policies place roughly a third of their recommendations on actions with negative estimated effect, against 15.5% for ours.

Comparing our policy against its own starting point isolates the contribution of GRPO from that of the base model. Training raises the estimated lift from 0.0170 to 0.0228, a relative improvement of 34% (Welch t-test over the five run-level estimates, p = 0.0009; because all policies are evaluated under the same five seeds, a paired test on the same estimates gives p = 0.003), while cutting the downside rate from 0.194 to 0.155 (p = 0.002) and improving tail risk from −0.094 to −0.073 (p = 0.002). The gain is therefore not merely a property of the base checkpoint: GRPO with the safety-gated rubric reward moves the policy toward actions with higher estimated effect and away from actions with negative estimated effect, which is precisely the behavior the safety gate was designed to induce.

The two evaluations do not rank identically. The audit and the judge agree that our GRPO policy is best, but they disagree elsewhere, and the disagreement is informative. The untrained Qwen3.5-27B base model ranks last on the judge rubric (8.457) yet second on the causal audit (0.0170), above every commercial system. Conversely, Claude Opus 4.5 scores well on the rubric (8.982) but has a negative estimated lift. We read this as evidence that the two evaluations measure genuinely different things. The rubric rewards well-formed advice: concrete figures, named entities, quantified impact, acknowledged risk. The audit rewards advice that maps to actions historically associated with gross-profit growth, and is indifferent to how well that advice is written. Commercial models are strong at the former and only middling at the latter, whereas the base model produces less polished recommendations that nonetheless land on reasonable actions. This divergence is the clearest available evidence that the audit is not a proxy for the judge; had it simply re-scored rubric quality, the base model would have ranked last on both. It also means the joint result carries more weight than either evaluation alone: our policy is the only system that ranks first under both.

Three findings stand out. First, GRPO with a finance-specific reward and explicit safety gate lifts an open-weight model above strong commercial LLMs on the judge rubric. This indicates that targeted reward design can substitute for raw scale on a narrow, high-value task. Second, the causal audit, which is independent of the judge, moves in the same direction: the GRPO policy attains a higher estimated gross-profit lift, a lower downside rate, and a less negative tail risk than every commercial baseline, with non-overlapping confidence intervals throughout. Because the audit is computed from logged outcomes and does not rely on the reward judge, this agreement makes pure judge adaptation a less likely explanation for the gains, although it does not establish realized impact. Third, and less expected, the two evaluations rank the other systems quite differently. That the untrained base model is last on the rubric but second on the audit shows the audit is not a restatement of the judge, and it also cautions against reading either evaluation as a complete measure of advice quality on its own: a policy can produce recommendations that are well-formed but point at unhelpful actions, or poorly-formed but point at sensible ones. Only a system that scores well on both, as ours does, is supported by both lines of evidence.

The safety gate and the risk-aware metrics are complementary. The gate suppresses harmful recommendations during training, while the downside rate and tail-risk metrics verify after training that the resulting policy places less mass on actions with negative estimated effects; the drop in downside rate from 0.194 (base) to 0.155 (ours) is direct evidence that this mechanism operates as intended.

We formulate financial advice generation as a reinforcement learning problem and train an open-weight model with GRPO using a finance-specific reward that incorporates explicit safety constraints. The resulting model outperforms the strongest evaluated commercial baseline on both the judge rubric and the CATE audit, and is the only system to rank first under both. It achieves approximately twice the estimated gross-profit lift while maintaining the lowest downside rate and least negative tail risk of any policy evaluated, and improves over its own starting checkpoint by 34% in estimated lift. We also find that the judge and the causal audit rank the baselines differently, which suggests that rubric quality and estimated business value are distinct axes and that reporting only one of them can be misleading. These results suggest that careful reward design, combined with a judge-independent causal audit to reduce the risk of reward hacking, provides a practical approach for building safer and more effective financial advice models. Future work will explore richer action taxonomies, evaluation by financial experts, and support for multi-step advice.

Because randomized experiments are not available in this setting, our causal audit is computed from logged observational data, in common with off-policy evaluation more broadly. We therefore treat it as a judge-independent, complementary signal alongside the rubric score rather than as a standalone measure, and the two evaluations are designed to be read together. A second limitation is coverage. Recommendations that receive no confident match in the 60-action catalogue are excluded from the audit rather than forced onto an ill-fitting action. This keeps the mapping conservative, but it means each policy is audited on the subset of its own output that the catalogue can express, and a policy whose unmatched recommendations differ systematically in quality from its matched ones would be scored on a non-representative subset. A richer catalogue would narrow this gap. Our audit also scores a single recommended action per business state; extending it to multi-step advice, where actions compound over a longer horizon, is a natural direction for future work. Finally, our study targets gross profit as the primary goal metric; while the framework supports other financial KPIs, we leave a broad multi-KPI evaluation to future work.

Improvements to AI Systems:

1. Implement RL-based fine-tuning with safety-gated, rubric-driven rewards.

• Use Group Relative Policy Optimization (GRPO) instead of supervised fine-tuning to avoid propagating suboptimal historical decisions.

• Design a multi-criteria reward rubric (e.g., 11 binary dimensions: safety gate, quantified impact, risk acknowledgment) scored by an LLM judge, with hard zeroing for harmful actions and penalties for unparseable or overly long outputs.

• The improved system can generate open-ended, actionable advice that is both well-formatted and explicitly avoids dangerous recommendations (e.g., eliminating primary revenue streams).

2. Integrate a judge-independent causal audit for off-policy evaluation.

• Map free-text recommendations to a fixed action taxonomy via a separate LLM classifier, then apply a doubly-robust (AIPW) estimator on logged observational data to estimate conditional average treatment effects (CATE) on a target KPI (e.g., gross-profit growth).

• Report lift, downside rate, and tail-risk (CVaR0.10) alongside judge scores.

• The improved system can detect reward hacking (where a model optimizes for judge appeal rather than real business value) and rank policies by genuine estimated impact, not just stylistic quality.

3. Adopt dual-evaluation protocols with divergence analysis.

• Always evaluate trained models on both an LLM-judge rubric and a causal audit, and explicitly compare rankings.

• Use divergence between the two (e.g., base model ranking last on judge but second on audit) as a diagnostic signal to identify whether a policy is overfitting to the judge or producing poorly-written but effective advice.

• The improved system can provide more trustworthy model selection in high-stakes domains, avoiding over-reliance on a single evaluation metric.

4. Enhance safety and risk-awareness through training-time gates and post-training verification.

• Embed a hard safety gate in the reward function that zeros out any recommendation with significant business harm (e.g., unsustainable cash-flow risk).

• After training, verify that the policy reduces the fraction of recommendations mapping to negative-effect actions (downside rate) and improves worst-decile performance (tail risk).

• The improved system can produce advice that is not only higher-performing on average but also more robust in worst-case scenarios, reducing the likelihood of catastrophic recommendations.

5. Improve generalization and coverage via richer action taxonomies and multi-step support.

• Expand the fixed action catalogue to cover more business actions, reducing the fraction of unmatched recommendations that are excluded from auditing.

• Extend the framework to multi-step advice, where actions compound over time, and to multiple KPIs (e.g., revenue, liquidity) rather than a single goal metric.

• The improved system can handle more diverse business contexts and provide sequential, long-horizon recommendations with validated causal impact across multiple financial objectives.

6. Use KL-anchored policy optimization to prevent drift.

• Add a KL penalty (weighted by β) to the GRPO objective to keep the fine-tuned policy close to the base model, avoiding catastrophic forgetting or degenerate outputs.

• The improved system can maintain stability during training while still achieving large performance gains (e.g., 34% lift improvement over base).

Generating actionable financial advice from business records demands that models integrate numerical reasoning, domain knowledge, and sound judgment, while avoiding recommendations that could harm the business. Direct supervision is difficult: historical decisions are not necessarily optimal, and high-quality free-form labels are expensive to obtain. We formulate financial advice generation as a reinforcement learning problem and fine-tune an open-weight language model using Group Relative Policy Optimization (GRPO). Our reward is an LLM-as-a-judge rubric that scores each recommendation across multiple binary dimensions of advice quality, augmented with a safety gate for harm prevention. Since LLM-based evaluation alone cannot confirm whether improvements reflect genuine business value rather than adaptation to the judge, we complement it with a judge-independent audit based on a standard doubly-robust Conditional Average Treatment Effect (CATE) estimator. Under this observational off-policy audit, our trained LLM achieves approximately twice the estimated gross-profit lift of the strongest evaluated commercial baseline (0.0228 vs. 0.0104), together with the lowest downside rate and the least negative tail risk of any policy evaluated. Notably, the two evaluations do not rank the baselines identically: the untrained base model places last on the judge rubric but second on the causal audit, indicating that the audit captures a signal the judge does not. Our results demonstrate that GRPO with a finance-grounded reward signal can produce substantially more useful business recommendations than commercial LLMs, and that a judge-independent causal audit is a valuable complement to, rather than a confirmation of, LLM-as-a-judge assessment in financial NLP.