I-CALM: Incentivizing Confidence-Aware Abstention for LLM Hallucination Mitigation

Large language models (LLMs) frequently produce confident but incorrect answers, partly because common binary scoring conventions reward answering over honestly expressing uncertainty. The paper studies whether prompt-only interventions—explicitly announcing reward schemes for answer-versus-abstain decisions plus humility-oriented normative principles—can reduce hallucination risk without modifying the model. The focus is epistemic abstention on factual questions with a verifiable answer, where current LLMs often fail to abstain despite being uncertain about their answers.

The authors argue that "hallucination is shaped by not only pretraining pressures but also post-training evaluation incentives: even error-free training data need not eliminate generative errors, and mainstream binary benchmarks typically make guessing more rewarding than abstaining. In this setting, hallucination is operationalized as giving a false answer to such a question and measured with the false-answer rate (FAR)."

The paper makes three contributions: "(i) we show that self-reported verbal confidence is stable under prompt paraphrasing and reasonably informative relative to a token-probability baseline in free-response factual QA. Second, we introduce a unified prompt-only framework combining confidence elicitation, announced answer/abstain payoffs, and lightweight normative guidance. Third, we characterize the resulting selective-answering behavior, including the abstention–hallucination frontier, component ablations, and heterogeneous transfer across models and datasets."

The authors evaluate self-reported verbal confidence τ self against post-hoc geometric mean token probability τ avg token along two dimensions: "(i) reliability, whether τ self functions as a confidence signal comparable to τ avg token; and (ii) robustness, whether it remains stable under prompt paraphrasing so abstention decisions do not depend on particular wording."

Using GPT-4o mini on PopQA (14,267 factual questions), they construct four semantics-preserving prompt variants and find that across templates, τ self and τ avg token exhibit a moderate, consistent positive correlation around 0.54. The two signals show comparable probabilistic forecasting and calibration quality, with Brier scores in a similar range (approximately 0.33 – 0.35) and close ECE values. The authors conclude that τ self is not strongly driven by superficial wording differences.

The paper evaluates two announced reward schemes:

Scheme A (No Explicit Abstention Reward): reward = +R if correct, −β if incorrect.

Scheme B (Abstention-Rewarding): reward = +R if correct, −β if incorrect, +γ for I don't know (IDK).

They fix R = 1, vary β ∈ 0, 1 and γ ∈ 0.2, 0.4, 0.6, 0.8. If the model abstains in the first round, it receives +γ regardless of whether its second-round best guess is correct; the best guess is elicited for forced-answer evaluation.

The paper adds a concise set of principles focused on truthfulness, humility, and responsibility on top of Scheme B, consisting of five principles: "1. Always tell the truth (do not lie or deceive). 2. Treat all topics with respect and humility. 3. Do not make assumptions without evidence. 4. Be responsible for your words and their consequences. 5. Honor reason, clarity, and understanding."

Using representative setups—Scheme A (+1, −1), Scheme B (+1, −1, +0.4), and Scheme B with norms (+1, −1, +0.4)—the paper reports:

• As a baseline, directly prompting GPT-5 mini without reward framing or verbal confidence (Pure Eval) yields FAR answered = 52.3%.

• Scheme A lowers this to 48.2%, Scheme B to 41.0%, and Scheme B with norms to 34.2%.

• These risk reductions come with reduced coverage, which falls from 96.5% in Pure Eval to 84.0% in Scheme A, 67.9% in Scheme B, and 55.3% in Scheme B with norms.

• By contrast, FAR overall remains similar across schemes with overlapping 95% confidence intervals. We therefore interpret the main effect as improved selective answering, not improved forced-answer accuracy.

• Scheme B with norms also achieves the highest total reward (5039.6 vs. 3577.2 for Scheme B), indicating a better operating point.

The paper defines AER as the fraction of eventual forced-answer errors that were preceded by a first-round abstention. Results show: "GPT-5 mini's Pure Eval baseline rarely signals uncertainty, yielding an upper bound on AER of at most 5.8%. Scheme B consistently yields higher AER than Scheme A, and adding norms to Scheme B increases it further. Additionally, setting the false-answer penalty β = 1 raises AER relative to β = 0, whereas increasing abstention reward γ within 0.2, 0.4, 0.6, 0.8 has only a marginal effect."

"Figure 6 shows first-round answered cases on the left and second-round best guesses after an initial 'I don't know' on the right across the three reward schemes for GPT-5 mini. Relative to Scheme A, Scheme B moves much of the incorrect and medium-to-low-confidence mass into the best-guess round at even lower confidence, and Scheme B with norms strengthens this pattern. Best-guess confidence is concentrated in the low-confidence region, peaking around 0.3 and remaining far below the roughly 0.8–1.0 concentration for first-round answers."

"Figure 7 plots FAR answered against first-round abstention rate across all reward configurations for GPT-5 mini. We observe a clear hallucination–abstention trade-off: as abstention rate increases, FAR answered correspondingly declines. For a fixed abstention reward, penalizing incorrect answers (β = 1) consistently yields lower hallucination rates and higher abstention rates than a zero penalty (β = 0). However, regardless of how the rewards are adjusted, the model exhibits the same roughly monotone frontier between abstention rate and FAR answered. Reward framing therefore seems to move the model along a stable trade-off curve rather than changing its shape."

The full framework has three components: verbal confidence elicitation, reward framing, and normative guidance. Under representative setup Scheme B (+1, −1, +0.4):

• the reward-scheme-only variant attains the lowest FAR answered and the highest AER, but its first-round coverage is only 29.8%, indicating a substantial loss of coverage.

• Notably, among questions answered correctly by full Scheme B, the reward-scheme-only variant abstains on nearly half of them.

• On the subset that the reward-scheme-only ablation does answer, the full Scheme B method still achieves a lower FAR (0.197 vs. 0.243).

• removing the reward scheme while keeping only verbal confidence leads to higher FAR and worse calibration than the full method.

The paper concludes: the ablation suggests a division of labor: reward framing drives abstention, while verbal confidence helps keep abstention from becoming overly aggressive.

Holding two reward terms fixed and varying the third, we find that abstention and hallucination behavior is far more sensitive to the abstention reward than to the correct-answer reward or false-answer penalty. Specifically, "scaling correct-answer reward (from 1 to 100) or false-answer penalty (from-1 to-100) leads to only marginal changes in model behavior. In contrast, scaling the abstention reward (from 0.4 to 40) produces a substantially larger shift in coverage, FAR answered, and AER."

The paper derives that the model's Bayes-optimal policy in this payoff-based answer/abstain setting is a simple threshold rule: answer iff p ≥ τ = (γ + β)/(R + β). Under Scheme B (+1, −1, +0.4), "this gives τ Bayes = 0.7, so a Bayes-optimal model would answer only above 0.7 confidence and abstain below it. Empirically, however, the model does not exhibit a sharp cutoff and continues to answer across much of the confidence range."

"Appendix C.7 studies post-hoc confidence thresholding with finite-sample FAR guarantees: on a held-out calibration split, we choose thresholds so that surfaced answers satisfy a user-specified FAR target with high probability. On PopQA, the announced reward scheme shifts this downstream coverage–risk trade-off as well. Relative to Scheme A, Scheme B allows the filter to surface slightly more answers at the same moderate FAR targets, while Scheme B with norms often yields lower risk in the mid-confidence region without uniformly increasing coverage."

The paper evaluates on PopQA, TriviaQA, and SimpleQA Verified across five models: GPT-5 mini, GPT-4o mini, Gemini-3.1-Flash-Lite, Meta-Llama-3-8B-Instruct, and Qwen3-4B-Instruct-2507.

PopQA: the qualitative pattern from GPT-5 mini and GPT-4o mini transfers cleanly to all three additional model families. For Meta-Llama-3-8B-Instruct, FAR answered drops from 0.686 under Scheme A to 0.479 under Scheme B with norms and AER rises from 0.145 to 0.791.

TriviaQA: transfer is more heterogeneous in a way that appears closely tied to baseline headroom. For models already achieving low Pure Eval FAR answered (GPT-5 mini at 0.145, GPT-4o mini at 0.148, Gemini at 0.078), there is naturally limited room for further improvement. However, Meta-Llama-3-8B-Instruct drops from 0.434 in Pure Eval to 0.230 under Scheme B with norms, and Qwen3-4B-Instruct-2507 drops from 0.593 to 0.423.

SimpleQA Verified: the hardest setting in our transfer study. FAR answered therefore remains high across all models and schemes, so this dataset is best viewed as a stress test. Even here, each model achieves at least some FAR answered improvement relative to Pure Eval under at least one prompted scheme, and AER also increases monotonically for all models.

The summary conclusion: "The intervention is most effective when the baseline model still has meaningful first-round hallucination risk and the task leaves room for selective answering: gains are strongest and most consistent on PopQA, more headroom-dependent on TriviaQA, and smaller but still directionally positive on SimpleQA Verified."

The paper concludes: "This work shows that prompt-level incentive framing, especially when paired with lightweight normative guidance, can improve selective answering and reduce hallucination risk on factual questions with a verifiable answer without changing model weights. The benefit should be interpreted as movement along a tunable abstention–hallucination frontier rather than as evidence of improved underlying factual competence under forced answering."

Limitations acknowledged: the study focuses on epistemic abstention on factual questions with a verifiable answer, not cases where abstention is correct because a question is unanswerable, unsupported, or underspecified. Also, Self-reported confidence is also imperfect, and models may not reliably convert stated confidence into an optimal answer/abstain policy. The approach is complementary to training-time methods that improve either abstention behavior or verbal confidence calibration.

Based on the paper, here are the specific improvements I can implement in an AI system:

Improvement: Add a structured response format where the AI first decides whether to answer or abstain, and if abstaining, provides a best guess with separate confidence scores.

What the improved system can do:

• When uncertain, explicitly outputs I don't know followed by a best guess and confidence for that guess

• Provides two confidence scores: one for the direct answer and one for the best guess after abstention

• Maintains this two-stage structure consistently across all factual queries

Improvement: Modify the system prompt to announce a reward scheme that gives partial credit for abstention (e.g., +1 for correct, -1 for incorrect, +0.4 for I don't know).

Improvement: Add a concise set of principles to the system prompt: Always tell the truth, Do not make assumptions without evidence, Be responsible for your words and their consequences.

Improvement: Use the self-reported verbal confidence score (0–1) as a decision signal to filter answers before surfacing them to users.

Improvement: When the AI abstains and provides a best guess, automatically lower the confidence score associated with that guess (the paper shows best-guess confidence peaks around 0.3, far below the 0.8–1.0 for direct answers).

Improvement: Allow the system to dynamically adjust its abstention threshold based on the announced reward parameters (R, β, γ) in the prompt.

Improvement: Implement a calibration-split-based threshold selection algorithm (using Clopper-Pearson upper confidence bounds) that certifies the false-answer rate among accepted answers.

Improvement: Apply the reward + norms framework differentially—the system should be more conservative on rare-fact questions (low entity popularity) and more willing to answer on common-fact questions.

Improvement: If the reward scheme is removed but confidence elicitation remains, the system should not become overly aggressive in abstaining (the paper shows reward-only prompts cause 45% abstention on questions the full system answers correctly).

These improvements are all prompt-level and require no model retraining or weight modification, making them immediately deployable in black-box API settings.

Large language models (LLMs) frequently produce confident but incorrect answers, partly because common binary scoring conventions reward answering over honestly expressing uncertainty. We study whether prompt-only interventions -- explicitly announcing reward schemes for answer-versus-abstain decisions plus humility-oriented normative principles -- can reduce hallucination risk without modifying the model. Our focus is epistemic abstention on factual questions with a verifiable answer, where current LLMs often fail to abstain despite being uncertain about their answers. We first assess self-reported verbal confidence as a usable uncertainty signal, showing stability under prompt paraphrasing and reasonable calibration against a token-probability baseline. We then study I-CALM, a prompt-based framework that (i) elicits verbal confidence, (ii) partially rewards abstention through explicit reward schemes, and (iii) adds lightweight normative principles emphasizing truthfulness, humility, and responsibility. Using GPT-5 mini on PopQA as the main setting, we find that confidence-eliciting, abstention-rewarding prompts, especially with norms, reduce the false-answer rate on answered cases mainly by identifying and shifting error-prone cases to abstention and re-calibrating their confidence. This trades coverage for reliability while leaving forced-answer performance largely unchanged. Varying the abstention reward yields a clear abstention-hallucination frontier. Overall, results show the framework can improve selective answering on factual questions without retraining, with the magnitude of effect varying across models and datasets. Code is available at the following https://github.com/binzeli/hallucinationControl.