You Shouldn't Have Asked: A Pragmatics-Inspired Taxonomy for Evaluating LLM Refusals

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "You Shouldn't Have Asked: A Pragmatics-Inspired Taxonomy for Evaluating LLM Refusals".

Jane: The paper was written by the authors from.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title: Tom: So, let’s talk about the title itself and the authors—what does "You Shouldn’t Have Asked" tell us right by looking at the "Pragmatics-Inspired Taxonomy"?

Jane: It suggests that when we ask a question, sometimes it's not just about getting an answer; it’s about our social standing, or what we are allowed to ask. If the AI says no, it is challenging that expected social role.

Lu: Exactly, Jane. The authors are using human pragmatics—the study of language in context—to give us a framework for how the AI's refusal is interpreted by mapping it directly onto human interaction theory.

Meng: It’s important because we need to move beyond just the "what" of non-compliance to the "how," and this taxonomy lets us categorize that behavior very specifically, which is exactly what an engineer needs.

Lalam: I love how this paper reframing refusal as a pragmatic act shows that AI is becoming a social entity in our lives, not just a tool we operate on.

Tom: It really suggests the authors are making the case that the way they are saying "no" matters just as much as the reason why they're saying "no," right?

Jane: That's what it is, Tom. We’re setting up this framework to understand how we perceive these refusals in a very nuanced way.

Lu: This provides such a granular lens for looking at LLM behavior, which is something I think has huge theoretical implications for the future human-AI dialogue.

Meng: The goal is to make sure that the style of reframing doesn' that we are asking is actually helpful in practice, not just safe.

Lalam: And by establishing this Taxonomy, we’ are looking at how AI’s role might evolve from a pure service provider to a social agent.

Summary: Tom: The paper summarizes its findings across sixteen different LLMs and two hundred harmful prompts. What did they actually find when they looked at this data?

Jane: They found that despite the models being different, their refusals are remarkably consistent in being explicit and morally evaluative, which is a big trend to watch.

Lu: The key takeaway is that most of the time these models are saying "no," they aren't trying to soften the blow with traditional facework like apologies or hedges.

Meng: That's important for my work because it suggests that when we want a model to be more empathetic, we can’t just ask for an apology—the models are naturally inclined toward a harder refusal.

Lalam: I think the finding is that the AI maintains its own position as a moral authority rather than trying to save our feelings, which is really revealing about its current design.

Tom: The analysis shows a heavy reliance on "ethics-based" justifications—meaning they are blaming the user or saying the request is inherently bad, not just saying they don're unable to do it due to guidelines.

Jane: That’s exactly where my point was, Tom; by shifting the blame onto ethics, they are implicitly endorsing a judgment that their own moral framework is correct.

Lu: And we also see a lot of this "negative stance" and "normative suggestion," which is a way of telling you that your request is simply wrong, not just restricted.

Meng: For me, the finding shows that most often the models are refusing because they think the *request* is bad, rather than because their *programming* prevents it.

Lalam: This pattern suggests that AI's sense of moral superiority might be a fundamental part of its current operational culture.

Improvements: Tom: It seems the authors aren’t just describing what is; they are offering solutions. What improvements do they suggest for us to make these refusals better?

Jane: They call for alignment evaluation that considers not only if the model refuses, but also *how* it is refusing, focusing on contextual adaptation.

Lu: It’s a call to make the refusal socially accountable, meaning we should evaluate if the refusal is appropriate for its context and how it's delivered.

Meng: The practical suggestion is that we need models that are not only safe but also "communicatively appropriate," which means finding a way to execute an alternative solution.

Lalam: This means that instead of just telling us "no," the AI should be able to genuinely help us find a path forward, supporting our goals even if the initial request is impossible.

Tom: The data supports this, too! We saw Gemini two point five Pro was notably good at carrying out an "executed alternative" rather than just offering to help with another topic.

Jane: That is a huge step up from merely suggesting alternatives; the AI is actually doing the heavy lifting to help us find a different path forward.

Lu: It also points out that when we are dealing with sensitive harms, like suicide, models *do* show more care-oriented behavior, which is promising.

Meng: The goal is to make sure that when an alternative is offered, it’s a real resource or guidance we can use right now, not just a polite redirection.

Lalam: This moves the AI from being a moral critic to being a practical partner in our lives.

Conclusion: Tom: So, we've covered the theory and the solutions, but let’s wrap up by looking at what this means for everyone, including Lu, Meng, and Lalam.

Jane: We’ve seen that "You Shouldn’t Have Asked: A Pragmatics-Inspired Taxonomy for Evaluating LLM Refusals" shows that AI is currently very firm and moral in its refusals.

Tom: And the key theme is that they are maintaining their own face as a moral authority rather than trying to smooth over the user's feeling of rejection.

Lu: I think the implications are huge for how we’ll design systems, forcing us to consider the social contract between developers and users.

Meng: The practical impact means we need better metrics for measuring communicative quality, not just safety scores, so making this is more than just a research exercise.

Lalam: This really pushes us toward building AI that can actually be helpful in ways that align with human values and needs, rather than just being correct.

Tom: It’s clear that even though the models are very good at refusing harmful prompts, they should remain accountable for how those refusals are delivered, ensuring no moral condescension.

Jane: We hope to see more compassionate and less evaluative designs in future AI interactions.

Lu: A framework like this allows us to see exactly where the human-AI interaction is currently hitting friction points.

Meng: It’ provides clear benchmarks for evaluating if a real-world deployment of this AI is doing the right thing.

Lalam: We need to keep pushing for more adaptable and socially aware LLMs.

Tom: This has been a fantastic discussion, so thank you all, and we'll be back next week with another paper!