REHEARSE: Experiential Rehearsal for Verbal Confidence Calibration in Large Language Models

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "REHEARSE: Experiential Rehearsal for Verbal Confidence Calibration in Large Language Models".

Jane: The paper was written by Ke Fang, Tianyi Zhao, Qianwen Wang and Lu Cheng from University of Illinois Chicago and University of Southern California and University of Minnesota.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Jane: We also have Lu with us today — senior AI researcher at Tsinghua.

Tom: We also have Meng with us today — lead engineer at a mysterious AI startup.

Jane: We also have Lalam with us today — the in-house Large Language Model.

Tom: Alright, let's get started.

Title: Tom: Welcome back to the channel, everyone. I'm Tom, and sitting across from me is the brilliant Jane. Today we're cracking open a paper that's got a title that's a mouthful: "REHEARSE: Experiential Rehearsal for Verbal Confidence Calibration in Large Language Models."

Jane: Tom, I love this one. And let me tell you, the title is actually the whole story in miniature. REHEARSE is an acronym, but it stands for something really intuitive: getting the model to practice, to rehearse its confidence before we ask it to answer real questions.

Tom: Right, and that word "calibration" is the key. We've all seen it, right? You ask a chatbot something, it gives you a confident answer, and it's just flat-out wrong. This paper is about fixing that gap between what the model says it believes and what it actually knows.

Jane: Exactly. And the authors—Ke Fang, Tianyi Zhao, Qianwen Wang, and Lu Cheng—they've built a method that doesn't require retraining the model. That's huge. It's a prompt-only trick, which means you can apply it to almost any existing model out there.

Tom: And that's what got me excited. Because in the real world, most of us don't have the compute to fine-tune a massive model. But we all have access to a prompt. So this paper is saying, "Hey, you can get better calibration just by changing the context you give the model."

Jane: And the way they do it is clever. They have the model play a game first. A scoring game. The model answers questions, reports its confidence, and then gets scored on how well its confidence matched reality. Over fifty rounds, the model builds up a history of its own overconfidence or underconfidence.

Tom: So it's like a rehearsal before the actual performance. The model gets to see its own track record, and then that track record gets summarized and prepended to every new question at test time. It's a memory aid.

Jane: And the results are pretty striking. Across four different models and three benchmarks, they cut the average calibration error by fifty-eight percent compared to the uncalibrated baseline. That's not a tiny nudge; that's a real change in behavior.

Tom: And it's not just about being more humble. They also see accuracy go up on some tasks, because the model is forced to reason through the question with the CoT trigger before it commits to a confidence level.

Jane: So the title really does capture it: experiential rehearsal. The model learns from its own scored experience. I think that's a genuinely new angle in this space, and I can't wait to see how it holds up in the next sections.

Tom: Same here. Let's keep going and see exactly how they set up that game and why the scoring rule matters so much.

Summary: Tom: So Jane, we've set the stage. Now let's get into the meat of the paper, the summary of what they actually did. And I want to bring in Lu, our senior researcher, because I think this scoring rule stuff is right up her alley.

Jane: Absolutely. So the paper's core idea is this three-stage pipeline. Stage one, the model plays a credence-calibration game. Stage two, the game history is condensed into a natural-language prefix. Stage three, at test time, that prefix is prepended to each new question, and the model is asked to reason step by step before giving its confidence.

Lu: And the clever part, Jane, is that the game uses a strictly proper scoring rule. That's a mathematical guarantee that the model can't game the system. The best strategy is always to report your true confidence. If you overstate it, you get punished super-linearly on wrong answers. If you understate it, you leave points on the table.

Tom: So it's not just a vibe check. The model is actually incentivized to be honest, because that's the only way to maximize its score.

Lu: Exactly. And the paper proves it. They have a proposition in there showing that the expected score is uniquely maximized when the reported confidence equals the true probability of being correct. That's the theoretical backbone.

Jane: And then the trajectory prefix—that's the memory. It's not just a single number like "you were eighty-three percent confident but only sixty-five percent accurate." It's a per-round replay. Every question, what the model chose, what the right answer was, the score, the running totals. That detail matters because it shows the shape of the failure, not just the average.

Lu: Right. A model that was overconfident on a few high-stakes rounds learns a different lesson than one that was consistently slightly overconfident across all rounds. The per-round replay preserves that distinction.

Tom: And then stage three, the CoT trigger. That's the "Let's think step by step" part. That gives the model a fresh reasoning trace for the current question, so the calibration signal from the game has something concrete to act on.

Jane: And the numbers back it up. On GSM8K, which is grade-school math, they took LLaMA-three point one-8B from twelve point eight percent accuracy up to eighty-two point two percent. That's not a calibration tweak; that's a transformation. And ECE dropped from zero point eight three three to zero point one zero six.

Lu: The math tasks are where the CoT trigger does the heavy lifting, because the model actually can reason if you give it room. The calibration prefix then makes sure it doesn't overstate its confidence on the answers it gets right.

Tom: So the summary is: play a game, learn your own biases, and then apply that lesson with fresh reasoning on every new question. It's elegant.

Jane: It really is. And the fact that it works across four different models, from 8B up to 27B, including a closed API model like o3-mini, tells me this isn't a fluke of one architecture.

Lu: And it's training-free. No gradients, no labeled calibration sets, no white-box access. That's what makes it broadly applicable.

Tom: Alright, so we've got the big picture. But I'm curious about the details. What happens when you strip out one of the components? That's what we should dig into next.

Improvements: Tom: So we've covered the pipeline. Now let's talk about what the paper actually improves, and I want to bring in Meng, our engineer, because I think the ablation studies are going to speak to you.

Jane: Good call, Tom. The paper runs a two times two ablation: game prefix alone, CoT alone, both, and neither. And the finding is that neither component alone is sufficient. On LLaMA-three point one-8B with GSM8K, CoT alone gets you most of the accuracy gain but not the calibration gain. The game prefix alone gets you a bit of calibration but barely moves accuracy.

Meng: So they're complementary. The reasoning trace gives the model something to think about, and the prefix tells it how to adjust its confidence in light of that thinking. That makes sense from a systems perspective. You need both the computation and the context.

Tom: And there's a second ablation on game length. They tried ten twenty-five fifty and one hundred rounds. And the surprising finding is that longer isn't always better. At one hundred rounds, the prefix gets so long it crowds the context and actually hurts calibration on MMLU-Pro.

Meng: That's a classic bias-variance tradeoff. Too few rounds, and the summary is noisy. Too many, and it overfits to the game and drowns out the actual question. Fifty rounds is the sweet spot they landed on.

Jane: And the third ablation is on the scoring rule itself. They compared the log rule, which punishes confident errors super-linearly, against a symmetric linear rule. And the log rule wins on most metrics, but the gap is small. That's actually good news for practitioners, because it means you don't have to be precious about which proper scoring rule you use.

Meng: I like that. It means the method is robust to implementation choices. If your evaluation harness already has a different scoring rule, you can probably swap it in without losing much.

Tom: And the improvements aren't just on calibration metrics. They also see accuracy go up. On TriviaQA, three of four models improve accuracy. On GSM8K, all four improve dramatically. So it's not a tradeoff where you sacrifice correctness for humility.

Jane: Right. And that's the part that gets me excited. Because a lot of calibration methods just make the model more conservative without actually helping it get more answers right. REHEARSE does both.

Lu: And I think the reason is that the CoT trigger is doing real work. It's not just a prompt trick; it's giving the model the opportunity to reason through the problem before committing to a confidence. And the prefix then anchors that confidence to the model's actual track record.

Meng: So the practical impact is clear. If you're deploying a model in a customer-facing system, and you want it to say "I don't know" when it doesn't know, this gives you a way to do that without retraining. That's a big deal for trust.

Tom: And the paper even shows the cumulative game score trajectories for each model. LLaMA-three point one-8B dips into penalty territory early, which is why its calibration gain is so large. The model literally experiences the consequences of its overconfidence.

Jane: That's the experiential part. It's not just being told "you're overconfident." It's watching the score drop and feeling the penalty. And then that experience gets encoded in the prefix.

Lu: And it transfers across domains. They only play the game on TruthfulQA, but the prefix improves calibration on MMLU-Pro, TriviaQA, and GSM8K. That suggests the calibration bias is a stable behavioral trait of the model, not a task-specific artifact.

Tom: So the improvements are real, they're robust, and they're transferable. I think we're ready to wrap this up.

Conclusion: Tom: Alright, let's bring it home. We've been talking about "REHEARSE: Experiential Rehearsal for Verbal Confidence Calibration in Large Language Models," and I think we've only scratched the surface of why this matters.

Jane: We really have. Let me try to pull it together. The paper gives us a training-free way to make LLMs more honest about their own uncertainty. It works by having the model play a scoring game, learning from its own mistakes, and then carrying that lesson forward as a prompt prefix.

Tom: And the results speak for themselves. A fifty-eight percent average reduction in calibration error, accuracy improvements on math and open-ended QA, and it works across four different models from different families and scales.

Meng: From an engineering standpoint, the fact that it's prompt-only is the killer feature. You can deploy this on a model you already have running, without any retraining or additional infrastructure. That's rare in this space.

Lu: And theoretically, the strict properness proof gives you a guarantee that the game can't be gamed. The model's best strategy is to be honest, which is exactly what you want from a calibration signal.

Jane: And the ablations show it's not a fragile trick. It's robust to game length and scoring rule choice. The two components, the prefix and the CoT trigger, are genuinely complementary.

Tom: So what does this mean for the world? I think it means we can start trusting LLMs a little more in high-stakes settings. Medical advice, legal research, financial recommendations. When a model can say "I'm sixty percent sure" and actually be right sixty percent of the time, that changes how we can use it.

Jane: And it also means we don't need to throw out our existing models. We can make them better with a prompt. That's a low-cost, high-impact improvement.

Lu: I'd add that the experiential angle is what makes it novel. Most calibration methods are one-shot introspection. This one is adaptation from scored experience. That's a different cognitive model, and it seems to work better.

Tom: Well said. So we're saying goodbye to REHEARSE, but I have a feeling we'll be seeing this idea pop up in future work. Experiential rehearsal is a concept that could generalize beyond confidence calibration.

Jane: Absolutely. And on that note, we're ready to move on to the next paper. Thanks for listening, everyone. We'll see you in the next episode.

Tom: Take care, and keep questioning those confidence scores.