Calibrating LLM Explanations to Human Decisions with Reinforcement Learning
summary
The gist
The gist: HD-RL, a reinforcement learning framework, calibrates an LLM explanation by optimizing it against observed human decisions to improve human decision-making quality.
In short
HD-RL is a reinforcement learning framework that optimizes Large Language Model (LLM) explanations by training them against observed human decisions. It calibrates a decision simulator based on human behavior to generate explanations that improve the quality of human decision-making, leading to significantly higher detection accuracy compared to other methods.
Key concepts
- Decision-Theoretic Framework
- This framework evaluates explanation quality by modeling how well it helps a person choose the best action. It trains a simulator on human data to predict the probability of decisions given a text and an explanation, using this prediction as a reward signal for optimizing the LLM.
- Decision Simulator (Sϕ)
- This model is trained on real human decisions to approximate how people will react to different explanations. It takes a text and an explanation as input and predicts the probability distribution of human choices. This simulator acts as a proxy reward model for the reinforcement learning process.
- Surrogate Reward (rϕ)
- Since directly optimizing expected utility is complex, HD-RL uses a log-transformed surrogate reward. This reward guides the LLM policy to favor explanations that lead humans to make correct decisions, while also penalizing explanations that seem unlikely based on the simulator's predictions.
- Explanation Format
- Explanations are structured as localized feature indicators, like 'Look for [specific linguistic or semantic feature]'. This format is chosen because it is quick for humans to read, refocuses attention on the original text, and never reveals the final decision to the user.
Terminology used across episodes
This episode discusses
- Calibrating LLM Explanations to Human Decisions with Reinforcement Learning · Paper Radio
- Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems
- To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making
- User Decision Guidance with Selective Explanation Presentation from Explainable-AI
- ComplLLM: Fine-tuning LLMs to Discover Complementary Signals for Decision-making
- How to Distinguish AI-Generated Images from Authentic Photographs
- Beyond Tools: Understanding How Heavy Users Integrate LLMs into Everyday Tasks and Decision-Making
- Measuring Faithfulness in Chain-of-Thought Reasoning
- RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
- AI Agents Push Humans Out of the Loop · Paper Radio
- Plausibility as Failure: How LLMs and Humans Co-Construct Epistemic Error
- "The explanation makes sense": An Empirical Study on LLM Performance in News Classification and its Influence on Judgment in Human-AI Collaborative Annotation
The paper
Calibrating LLM Explanations to Human Decisions with Reinforcement Learning · Read on arXiv
Dong Shu Ziyang Guo, Negar Kamali, Matthew Groh, Jessica Hullman
Northwestern University
Explanations of LLM outputs are considered an important way to make model behavior legible, helping human users make good decisions on downstream tasks. However, default training approaches for improving explanations target model task performance or human preference rather than human decision quality. To bridge this gap, we introduce HD-RL, a reinforcement learning framework for optimizing LLM explanations for human decision making. Our approach calibrates a human decision simulator on observed human decisions, applying a proxy reward model to optimize the LLM to generate explanations that help people to make good decisions. We apply this framework to AI-generated text detection. In our initial data collection phase, we observed that while LLMs possess strong autonomous detection capabilities, the explanations they generate for users yield near-random human accuracy (51.3%). By applying the HD-RL framework to the open-source Qwen 3.6 model, we generate optimized explanations that improve human performance, achieving a GLMM-estimated human accuracy of 63.7% in a randomized between-subjects experiment with repeated measures and outperforming RLVR, RLAIF, and other baselines by over 8%.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: Today's paper: "Calibrating LLM Explanations to Human Decisions with Reinforcement Learning".
Jane: The gist: HD-RL, a reinforcement learning framework, calibrates an LLM explanation by optimizing it against observed human decisions to improve human decision-making quality.
Tom: First, who's behind it and why it matters.
Paper summary: Tom: So if we boil it down for the listener on this paper, HD-RL is a way to train an LLM explanation policy to maximize the expected utility of human decisions induced by those explanations Tom. It’s about using observed human choices to build a simulator and then using that simulator as a proxy reward model Tom.
Jane: The implication is that for AI systems that interact with people, we need to ground their outputs in what actually helps people make good choices, rather than just making them look right on a benchmark or pleasing to the user Jane. It shifts the focus from model-centric metrics to human-centric outcomes Jane.
Lu: What this means for representation learning is that you can use external decision data, even if it's observational, to guide the explanation generation process towards a more useful distribution of explanations Lu. It’s about injecting real-world decision theory into the training loop.
Meng: From a deployment standpoint, this framework gives us a way to push an AI not just to be technically competent but also contextually effective in how it communicates its findings to end users Meng. That contextual effectiveness is something that usually gets ignored in standard RL setups.
Lalam: And they've released their human decision dataset and the trained model, which opens the door for other researchers to build on this work with more data or different tasks Lalam. It’s an open contribution to the field of explanation calibration.
Tom: So that’s it for this discussion on "Calibrating LLM Explanations to Human Decisions with Reinforcement Learning" Tom. We've covered what the paper is about, how they did it, and what this means for making AI explanations actually useful in collaboration with humans.
Conclusion: Tom: So, to recap, HD-RL is this reinforcement learning framework that takes an LLM explanation and calibrates it by optimizing it against what real people actually decide when they see that explanation Tom.
Jane: It’s essentially trying to make sure the AI explanations actually help humans make better choices on their own tasks instead of just being technically accurate or pleasing to the model Jane.
Lu: The core idea is training a decision simulator on observed human decisions and then using that simulator as a reward model to fine-tune the LLM policy Lu. It’s about grounding the explanation in human action utility Lu.
Meng: From what I see, they’re focusing on how to translate those messy human decisions into something the AI can actually use to guide its output generation Meng. It sounds like a practical step toward making AI more useful in real-world settings Meng.
Lalam: I'm pretty sure the authors are showing that this method significantly boosts human accuracy when dealing with text generated by an AI, outperforming other alignment methods by a noticeable margin Lalam.
Tom: That's good to hear about the results, Lalam. But what’s the actual takeaway for us as developers? It means we can start optimizing explanations not just for internal metrics but for actual human decision quality Tom.
Jane: Exactly. We need to move past just making sure the text looks plausible and start thinking about how that text steers the person reading it toward a better outcome Jane.
Lu: The design of the explanation format itself is also part of this work, using structured feature indicators like "Look for specific linguistic or semantic feature" which they say minimizes reading time and forces attention back to the text Lu.
Meng: That formatting choice makes sense from a deployment standpoint. If people have to spend too much time deciphering the explanation itself, it defeats the purpose of a quick decision-making tool Meng.
Tom: So, we've seen how they train this simulator and optimize the policy, and now we see the final outcome is higher human detection accuracy than other methods Tom.
Jane: It’s a big shift because it moves away from just measuring how well the AI matches some reference model to actually measuring how useful its output is for a person in a real situation Jane.
Lu: The authors are releasing their decision dataset and the trained model, which means this isn't just a theoretical concept; they’ve provided the tools for others to build on this work with more data Lu.
Meng: That open contribution is what matters. If we can use that training data and framework, we can start designing explanations that actually perform better when deployed at scale Meng.
Tom: So, in summary, HD-RL is a way to train LLM explanations using observed human decisions to directly improve the quality of those human decisions Tom.
Jane: It’s about making sure the AI's guidance is relevant to what people actually need it for Jane.
Lu: The next thing we should look at is how they handled the limitations, because every method has its stopping point Lu.
More episodes
- 2610.17132-Masked Attention Pooling Is Invariant to Padding and Permutation: A Formal Proof in Lean
- 2610.16226-NetRL: Powering Network Decisions via NetGRPO-Driven Reinforcement Learning
- 2610.17154-Neuromorphic Decoding and Dynamical Image Reconstruction from Populations of Spiking Retinal Ganglion Cells
- 2610.17018-LLM Evaluation is a Forecasting Problem
- 2610.17321-GPU Goldmines: Tunable WebGPU Kernels for Edge LLM Inference
- 2610.20655-Global-in-time Spectral Duhamel-Picard Method for Post-Exposure Bake Simulation and Physics-Informed Neural Operators for EUV Lithography
- 2610.17265-Protocol Choices Matter More Than the Aggregation Algorithm: Federated Learning for In-Hospital Mortality Prediction in Care-Unit and Multi-Hospital Federations
- 2610.20652-Does Personalization Break Trust? Open-Set Rejection Calibration Under Few-Shot On-Device Intent Adaptation
- 2610.14338-STREGO: A Scalable Implementation of a Trust-Region Framework for Noisy Bayesian Optimization
- 2610.12471-Neuromorphic Satellite Intelligence: A Formal Framework for Event-Driven Architectures, Spike-Based Algorithms, and Quantitative Energy-Accuracy Trade-offs