Is This Just Fantasy? Language Model Representations Reflect Human Judgments of Event Plausibility

summary

The gist

The gist: Language models have been shown to learn linear representations that discriminate between different modal categories, and these vectors reflect human categorization behavior and

In short

This research found that language models learn linear representations called 'modal difference vectors' to categorize sentences based on possibility (e.g., possible, impossible, inconceivable). These vectors are more reliable than simple probability estimates and reflect how humans actually categorize events. The study shows these vectors correlate with human judgments about event likelihood and features like imageability.

Key concepts

Modal Difference Vectors: These are linear representations discovered within language models that effectively separate different modal categories, such as possible versus impossible. They are created by calculating the difference between the hidden states of two stimuli belonging to distinct categories, allowing the model to discriminate between them more accurately than standard probability scores.
Modal Categories: The paper defines four levels of possibility: Probable (commonplace and possible), Improbable (possible but rare), Impossible (violates known laws like physics), and Inconceivable (fundamentally flawed or un-evaluable). These categories help map how language models structure the concept space of events.
Contrastive Activation Addition: This is the specific technique used to create modal difference vectors. It involves taking the difference between the hidden state representations of two examples that belong to different categories, such as one representing a 'possible' scenario and another representing an 'impossible' one. This subtraction yields a vector that highlights the key differences between those concepts.
Mechanistic Interpretability: This is the field of study used to understand how internal components of language models work. The researchers use this approach to analyze LM activations, treating them as interpretable features that reveal underlying principles, such as how models distinguish between different levels of modal categories.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 19 episodes
Federated Learning — explained in 17 episodes
Retrieval-Augmented Generation (RAG) — explained in 17 episodes
Quantum Fisher Information (QFI) — explained in 16 episodes
Interpretability — explained in 15 episodes

This episode discusses

The paper

Is This Just Fantasy? Language Model Representations Reflect Human Judgments of Event Plausibility · Read on arXiv

Michael A. Lepori, Jennifer Hu, Ishita Dasgupta, Roma Patel, Thomas Serre, Ellie Pavlick

Brown University · Johns Hopkins University · Google DeepMind

Language models (LMs) are used for a diverse range of tasks, from question answering to writing fantastical stories. In order to reliably accomplish these tasks, LMs must be able to discern the modal category of a sentence (i.e., whether it describes something that is possible, impossible, completely nonsensical, etc.). However, recent studies have called into question the ability of LMs to categorize sentences according to modality (Michaelov et al., 2025; Kauf et al., 2023). In this work, we identify linear representations that discriminate between modal categories within a variety of LMs, or modal difference vectors. Analysis of modal difference vectors reveals that LMs have access to more reliable modal categorization judgments than previously reported. Furthermore, we find that modal difference vectors emerge in a consistent order as models become more competent (i.e., through training steps, layers, and parameter count). Notably, we find that modal difference vectors identified within LM activations can be used to model fine-grained human categorization behavior. This potentially provides a novel view into how human participants distinguish between modal categories, which we explore by correlating projections along modal difference vectors with human participants' ratings of interpretable features. In summary, we derive new insights into LM modal categorization using techniques from mechanistic interpretability, with the potential to inform our understanding of modal categorization in humans.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Is This Just Fantasy? Language Model Representations Reflect Human Judgments of Event Plausibility".

Jane: The gist: Language models have been shown to learn linear representations that discriminate between different modal categories, and these vectors reflect human categorization behavior and interpretable features.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, wrapping up this whole discussion on "Is This Just Fantasy? Language Model Representations Reflect Human Judgments of Event Plausibility." The main point is that these linear representations, the modal difference vectors, are more reliable for classifying things than just looking at the probability scores.

Jane: And what matters is that these vectors show a consistent way models learn to build their understanding of possibility and impossibility as they grow larger and more complex > Page ten >

Lu: This research moves beyond just seeing if an LM can answer a question correctly; it shows us *how* the AI structures its knowledge about the world, whether it’s coarse or fine-grained > Page two > <ref:2507.12553#pg3>

Meng: For deployment at scale, this suggests that we need to pay attention to these internal representations because they actually reflect how humans think about things like event plausibility > Page ten >

Tom: And for someone just listening to the show, the implication is that the AI isn't just spitting out text; it’s building a structured, layered model of reality based on how it processes language.

Jane: It suggests we might be able to use these vectors to test intuitive theories about what LMs think about the world and see if they match human intuition > Page two > <ref:2507.12553#pg3>

Lu: The paper suggests that LMs encode causal principles underlying the world while retaining the ability to imagine hypothetical realities > Page ten >

Meng: So, we’re looking at a way to test those theories directly using these specific mathematical features from the models themselves.

Tom: That's what this work does, identifying linear representations that discriminate between modal categories within a variety of LMs, or modal difference vectors > Page one > <ref:2507.12553#pg2>

Jane: It really gives us a new lens into how we understand modal categorization in humans by correlating projections along these vectors with human participants' ratings of interpretable features > Page ten >

Conclusion: Tom: So, we’re wrapping up our look at this paper, "Is This Just Fantasy? Language Model Representations Reflect Human Judgments of Event Plausibility." The authors are Michaelov et al., and they found that these mathematical vectors actually align with how humans categorize possibilities.

Jane: It seems the big idea is that language models aren't just guessing; they’ve built these internal structures that separate what’s probable from what’s impossible, which matches our intuition about the world.

Lu: The core finding is that these modal difference vectors are more discriminative than the standard probability estimates we usually look at when classifying sentences.

Meng: From a deployment standpoint, it means we can actually use these specific internal features to classify text better than just looking at the raw output score, which is a nice mechanism for refinement.

Tom: Exactly. And what’s interesting is how these vectors emerge as the models get bigger and more trained; they follow a specific developmental order that mirrors how we think about different kinds of impossibility.

Jane: That’s where it gets really compelling for us—when we project those vectors onto human behavioral data, they actually correlate well with what people say is "subjective event likelihood."

Lu: The paper suggests that this might be a way to test intuitive theories about what an AI learns from just reading text. It gives us a concrete feature space derived directly from the model's activations.

Meng: But we need to watch how these vectors behave when the model starts failing at scale, because if they break down, we know where our representation learning needs to improve for production systems.

Tom: Right. So what does this actually mean for the future of AI understanding? Does this help us design better models, or does it just give us a fancy way to categorize text?

Jane: It gives us a new tool—a way to peek inside the model’s head—to see if its internal logic about reality matches our own experience of what's possible.

Lu: It suggests that LMs are encoding causal principles underlying the world while still keeping the flexibility to imagine hypothetical scenarios. That’s a pretty deep claim for representation learning.

Meng: If we can use these vectors to check if an AI is reasoning about physics or just random word patterns, that could make our safety testing much more rigorous when we deploy these systems.

Tom: So, the paper shows a specific way to look at model internal states and connect them back to human-understandable concepts like imagination and physical possibility. It’s a bridge between the code and common sense.

Jane: It really does point toward testing intuitive theories about what LMs derive from raw text input, which is something we need to keep exploring as these models get more sophisticated.

More episodes

← Home