From Plausible to Actionable: A Position on LLM Self-Explanations

arXiv:2607.15957 · cs.CL, cs.AI · Submitted 2026-07-17 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "From Plausible to Actionable: A Position on LLM Self-Explanations".

Jane: The paper was written by Elize Herrewijnen, Benedetta Muscato, Gizem Gezici and Fosca Giannotti from University of Utrecht, Utrecht, Netherlands and National Police Lab AI, Netherlands Police, Driebergen, Netherlands and Scuola Normale Superiore, Pisa, Italy - University of Pisa.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Summary: Tom: Now, let’s get into the core of what this paper summarizes. It makes a strong argument about the current state of AI evaluation methods when applied to these LLM outputs. They argue that while these self-explanations are often highly plausible, they are questionably faithful.

Jane: The paper argues that standard XAI protocols simply aren't built to handle how Large Language Models actually operate, which is a major roadblock for us. These traditional methods assume a direct relationship between the reasoning and the prediction, but that assumption breaks down with generative models.

Lu: It highlights three main reasons why we can’t just rely on those old evaluation standards. First, they point out that LLMs lack access to their internal processes when generating these explanations; they aren't introspecting on their own decision-making process in a genuine way.

Meng: And I think the issue of nondeterminism is huge too. We know that identical inputs might not produce identical outputs across multiple runs, and the explanation generation itself varies significantly, which makes a single reference output useless for me.

Lalam: It’s like trying to evaluate a musical composition by only listening to one performance when you know that subtle variations in tempo and dynamics are part of its inherent nature. The model’s ability to generate different explanations for the same task reflects that complexity.

Tom: So, it's not just that they fail; it's that current evaluation methods are fundamentally incompatible with the dynamic, unpredictable nature of LLMs.

Jane: Exactly, and the paper also brings up issues around how humans evaluate things. They noted that we need to consider both experts and lay users when assessing plausibility because a non-expert might find complex terminology convincing while a doctor recognizes flawed reasoning.

Lu: Furthermore, they point out that relying on just one single reference explanation doesn't capture the wide variation in how different people might explain an ambiguous or subjective task.

Meng: If we are building robust systems, we can’t depend on one fixed ground truth when the real world is full of multiple valid interpretations.

Lalam: We have to respect that human perspective while acknowledging that the AI's internal logic operates in a vastly different, more complex space entirely. This summary really sets up the problem we need to solve before moving forward.

Improvements: Tom: Given these fundamental flaws and limitations, what practical improvements does this paper suggest for "From Plausible to Actionable: A Position on LLM Self-Explanations"? It’s a pivot from criticism to solutions.

Jane: The biggest shift they propose is moving away from just measuring how plausible or faithful the explanation is, and toward prioritizing what they call "actionability." This changes the definition of success completely.

Lu: Actionability means judging if an explanation actually helps a stakeholder make a better decision or take concrete steps in the real life world. We are shifting from assessing theoretical logic to practical utility.

Meng: That’s extremely appealing from my side; I want output that translates directly into informed decision-making, not just some academic score on how similar the explanation is to another document.

Lalam: It moves the conversation away from "Is this prediction correct?" to "What does this enable us to do next with a human or a system" which is much more aligned with human needs.

Tom: The authors suggest specific ways that self-explanations can serve as communicative interfaces, making complex XAI techniques accessible to non-expert users. It’s about simplification without losing substance.

Jane: And they also propose using these explanations not just as final answers, but as tools to support human decision-making in high-stakes fields like medicine or legal review.

Lu: They advocate for treating LLMs as "advocates," meaning the model presents various complementary arguments and counterarguments, not definitive answers. This is a big change in how we view AI agency.

Meng: This approach helps mitigate the risk of automation bias because the system is designed to surface diverse perspectives, rather than just telling us what to do. It forces critical evaluation from a practical standpoint.

Lalam: By giving different viewpoints a platform, this method allows us to foster genuine deliberation by expanding our collective pool of information and refine our decisions. This concept really gives us something powerful to build on for the next phase of discussion.

Conclusion: Tom: So, let’s quickly revisit the main takeaway from "From Plausible to Actionable: A Position on LLM Self-Explanations." We've seen how this paper critiques our current evaluation methods.

Jane: We established that while LLM self-explanations are often highly plausible to humans, they are questionably faithful because of the internal complexities of the models.

Lu: The authors emphasize that relying solely on traditional metrics ignores the non-determinism and prompt sensitivity inherent in large language models when we try to measure faithfulness.

Meng: The path forward is operationalizing actionability—designing systems that actually support informed choice, rather than just generating a convincing narrative. This is how we make these tools useful.

Lalam: This paper is pushing us toward a future where an AI’s ability its to clearly articulate its reasoning is seen as a powerful tool for cultural advancement and deeper reflection on complex problems.

Tom: We have covered the limitations, the proposed improvements, and how this shifts our focus from pure faithfulness to actionable value. It's clear that the ultimate goal of making AI accessible is not just about making it look right, but about helping us make better decisions.

Lu: I think the idea that LLMs should be advocates is incredibly powerful; it opens up so many possibilities for complex problem-solving.

Meng: I’m excited about building guardrails that enforce this "argument-present" approach described in the paper to ensure we can actually implement these solutions practically.

Lalam: This allows us to integrate AI into human thought processes in a way that is truly collaborative and culturally enriching for everyone involved.

Jane: It's very hopeful to see a guide for moving past traditional metrics and toward focusing on what makes these systems genuinely useful for the listeners.

Tom: Thank you all for this incredibly insightful discussion about "From Plausible to Actionable: A Position on LLM Self-Explanations." We hope this gives our listeners something concrete to consider as they explore the future of AI and human decision-making.

Conclusion: Tom: So, we’ve really spent our time today dissecting "From Plausible to Actionable: A Position on LLM Self-Explanations" and looking at why traditional AI evaluation metrics are fundamentally unsuited for understanding these generative models.

Jane: It's clear that while the explanations often look convincing to humans, they aren't necessarily reflecting the true internal logic of the AI due to things like nondeterminism and complexity.

Tom: And that’s where the authors suggest shifting our entire focus toward actionability—making sure these explanations actually help us make better decisions, which is a huge leap from simple plausibility scores.

Lu: I think this shift towards actionable insights is incredibly exciting because it opens up so many new ways we can structure complex problem-solving workflows.

Meng: The idea of operationalizing actionability also ensures that the systems are practical, providing genuinely useful outputs rather than just some theoretical narrative we have to analyze later.

Lalam: For me, this means that as a cultural advancement, these tools allow us to foster a deeper level of deliberation in human thought processes.

Tom: I think we’ve covered all the critical aspects of "From Plausible to Actionable: A Position on LLM Self-Explanations" today and seen how this redefines the relationship between AI and human decision-making.

Jane: It’s a great framework for seeing beyond just what makes an explanation look right, focusing instead on what makes it truly useful.

Tom: I know you all have some final thoughts before we wrap up this segment of the show.

University of Utrecht, Utrecht, Netherlands · National Police Lab AI, Netherlands Police, Driebergen, Netherlands · Scuola Normale Superiore, Pisa, Italy - University of Pisa

cs.CL, cs.AI

Submitted: 2026-07-17

Updated: 2026-09-22

Comments: 5 pages

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 76/100

The gist: The emergence of Large Language Models (LLMs) has created new opportunities in explainable artificial intelligence (XAI), particularly through self-explanations—natural language explanations

Key concepts

Actionability: A proposed shift in evaluating LLM explanations, moving beyond theoretical logic. Actionability judges whether an explanation actually helps a stakeholder make a better decision or take concrete steps in the real world.
Self-Explanations: The process where Large Language Models (LLMs) generate explanations for their own outputs. The paper discusses how these explanations, while often plausible, are not necessarily faithful to the model's true internal reasoning.
Nondeterminism: The issue that identical inputs given to an LLM may not produce identical outputs across multiple runs. This variability means that relying on a single reference explanation is unreliable for evaluation.

Terminology

Actionability — 14× in this paper · explained in 3 episodes
Interpretability — 9× in this paper · explained in 15 episodes
Hallucination — 6× in this paper · explained in 4 episodes
Mechanistic interpretability — 5× in this paper · explained in 2 episodes
Alignment — 4× in this paper · explained in 1 episode
Large Language Models (LLMs) — 4× in this paper · explained in 6 episodes
Explainability — 3× in this paper · explained in 3 episodes
Robustness — 3× in this paper · explained in 5 episodes
Sycophancy — 3× in this paper · explained in 4 episodes
Accountability — 2× in this paper · explained in 2 episodes
Accuracy — 2× in this paper · explained in 1 episode
Adam — 2× in this paper · explained in 2 episodes
Chain-of-Thought Prompting — 2× in this paper · explained in 2 episodes
Generative AI — 2× in this paper · explained in 2 episodes
Generative Model — 2× in this paper · explained in 2 episodes
Generative Models — 2× in this paper · explained in 3 episodes
Benchmark — 1× in this paper · explained in 2 episodes
Benchmarking — 1× in this paper · explained in 4 episodes
Consistency — 1× in this paper · explained in 3 episodes
Core — 1× in this paper · explained in 2 episodes
Decision-Support Tool — 1× in this paper · explained in 2 episodes
Human-in-the-Loop — 1× in this paper · explained in 3 episodes
LLM-as-a-Judge — 1× in this paper · explained in 3 episodes
Prompt Engineering — 1× in this paper · explained in 4 episodes
Xplainable Artificial Intelligence — 1× in this paper

Summary

The emergence of Large Language Models (LLMs) has created new opportunities in explainable artificial intelligence (XAI), particularly through self-explanations—natural language explanations generated by the model for its own decisions. This position paper argues that while these self-explanations are highly plausible and accessible to stakeholders, their reliability is questionable. The authors propose a fundamental shift in evaluation, moving beyond the traditional metrics of plausibility and faithfulness to emphasize the practical value of actionability in supporting informed decision-making across diverse user groups.

Limitations in Assessing Plausibility

Current methods for evaluating plausibility often overlook critical factors related to human reasoning and task complexity. The paper notes that reliance on a single reference explanation risks collapsing diverse but valid perspectives into one standard. Furthermore, conventional evaluation protocols fail to account for the stakeholder’s varying levels of expertise. For instance, a doctor might judge an explanation differently than a lay user, meaning plausibility assessment must be collected and assessed according to the stakeholder’s specific background.

The Questionable Nature of Faithfulness

The authors argue that self-explanations are questionably faithful because standard XAI evaluation protocols are fundamentally unsuited for LLMs. The paper identifies three main reasons why faithfulness is difficult to verify:

LLMs lack access to the internal processes that generate their outputs, meaning self-explanations may not reflect genuine introspective analysis.
Prompting models to explain predictions can influence the decision process itself, potentially altering accuracy.
LLMs can provide coherent but misleading rationales due to sycophantic behavior, hallucinations, or biases.

Moreover, traditional perturbation-based methods fail because they rely on flawed assumptions regarding LLM behavior:

Prompt Sensitivity: LLMs are highly sensitive to subtle changes (e.g., spacing or capitalization) in prompts.
Model Reliance: An LLM may disregard the task input and instead rely solely on its internal knowledge, making perturbation-based signal assessment unreliable.

Shifting Focus to Actionability

Given the limitations of current evaluation frameworks, the paper advocates for a new focus on actionability—the ability of self-explanations to help stakeholders make informed decisions and take concrete actions. This perspective views self-explanations not as literal accounts of internal reasoning, but as communicative interfaces that allow XAI methods to be more accessible.

The authors outline three practical applications for this shift:

Making XAI Accessible: Translating complex, faithful explanations into natural language rationales that are easier for non-expert stakeholders to understand.
Supporting Human Decision-Making: Using self-explanations as arguments that human decision-makers can critically evaluate, rather than as definitive proof of a model's reasoning.
Facilitating Deliberation: Utilizing different LLMs to surface complementary arguments and counterarguments to enrich the decision-making process.

Improvements for AI systems

Based on the critical analysis of LLM self-explanations provided in this paper, the following architectural, evaluation, and deployment improvements are necessary to transition from merely Plausible models to genuinely Actionable AI systems.

The core of the improvement is replacing traditional XAI metrics with a comprehensive Actionability Framework.

Specific Improvements:

Mandatory Multi-Perspective Validation: Instead of seeking a single reference explanation, the system must be validated against diverse, context-aware explanations. Evaluation protocols must incorporate both task experts (e.g., doctors) and lay users. The system's performance is measured by its ability to satisfy the specific needs of each group, not just by agreement with one consensus.
Incorporating Deliberative Utility: A metric for Deliberative Utility must be established, quantifying how a generated explanation facilitates critical thought or counter-argument generation among human stakeholders.
Reject Single-Run Assumptions: Evaluation protocols must explicitly account for LLM non-determinism, requiring statistical validation across multiple runs rather than relying on single outputs to determine faithfulness.

What the Improved System Can Do:

The system can be certified as Actionable, meaning its utility is proven not just by its apparent correctness, but by its demonstrated ability to support complex human decision-making processes.

To address the question of whether self-explanations truly reflect the model's internal process, constraints must be imposed to prevent misleading rationales or sycophancy.

To solve the accessibility gap between complex XAI methods and non-expert stakeholders, the explanation delivery must be highly adaptive.

To leverage LLMs as advocates for human judgment.

Abstract

Large Language Models (LLMs) can generate natural language explanations that rationalize their own decisions, a phenomenon commonly referred to as self-explanations. Such explanations have emerged as a promising direction for explainable artificial intelligence (XAI), particularly for interpreting LLM behavior. However, while self-explanations often appear plausible, whether they faithfully reflect a model's underlying reasoning process remains an open question. In this opinion paper, we argue that self-explanations can be highly plausible, questionably faithful, and yet highly actionable. From a traditional XAI perspective, we identify the limitations of standard evaluation protocols for LLM-generated self-explanations and propose practical guidelines for assessing their plausibility and faithfulness.Moreover, we argue that evaluation should extend beyond these criteria to actionability, highlighting applications of LLM rationalization capabilities that support informed decision-making and appropriate action across diverse stakeholders.