The Trilemma of Truth in Large Language Models

summary

The gist

Given the high stakes of accuracy, I will synthesize these technical details into a comprehensive and detailed summary that captures the methodology, core findings regarding flawed assumptions, and

In short

This research introduces sAwMIL, a new multiclass probing framework to assess LLM veracity by classifying statements as true, false, or neither. It identifies flaws in existing methods that assume single axes and perfect ground truth. sAwMIL uses Multiple-Instance Learning and Conformal Prediction to find robust, transferable veracity directions that have causal links to model outputs.

Key concepts

sAwMIL (Sparse-Aware Multiple-Instance Learning): A novel multiclass probing framework that classifies LLM statements into three categories: true, false, or neither. It uses a three-stage training process involving MIL and Conformal Prediction to establish statistically guaranteed prediction sets, moving beyond simple binary truth/false assessments.
Three-Valued Logic Approach: The paper argues that veracity should be modeled using a three-valued logic (true, false, or neither) instead of just true or false. This approach allows the probe to accurately identify statements where the LLM simply lacks knowledge, assigning them to the 'neither' class.
Contrastive Token Intervention (CTI): A causal validation technique used to test if a learned veracity direction actually influences an LLM's output. CTI helps researchers determine if the identified veracity signal has a demonstrable causal effect on specific tokens or parts of the model's response.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Retrieval-Augmented Generation (RAG) — explained in 21 episodes
Sample Complexity — explained in 20 episodes
Flow Matching — explained in 19 episodes
Quantum Fisher Information (QFI) — explained in 19 episodes
Federated Learning — explained in 17 episodes

This episode discusses

The paper

The Trilemma of Truth in Large Language Models · Read on arXiv

Germans Savcisens, Tina Eliassi-Rad

Khoury College of Computer Sciences Northeastern University · Network Science Institute Northeastern University · Santa Fe Institute

The public often attributes human-like qualities to large language models (LLMs), assuming that they "know" certain things. In reality, LLMs encode information retained during training as internal probabilistic knowledge. This study examines existing methods for probing the veracity of that knowledge and identifies three flawed underlying assumptions. To address these flaws, we introduce sAwMIL (Sparse-Aware Multiple-Instance Learning), a multiclass probing framework that combines multiple-instance learning with conformal prediction. sAwMIL leverages LLMs' internal representations to classify statements as true, false, or neither. We evaluate sAwMIL across 16 open-source LLMs, including default and chat-based variants, using three new curated datasets. Our results show that (1) common probing methods fail to provide a reliable and transferable veracity direction and, in some settings, perform worse than zero-shot prompting; (2) truth and falsehood are not encoded symmetrically; and (3) LLMs encode a third type of signal that is distinct from both true and false.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "The Trilemma of Truth in Large Language Models".

Jane: Here's the gist of the paper. Given the high stakes of accuracy, I will synthesize these technical details into a comprehensive and detailed summary that captures the methodology, core findings regarding flawed assumptions, and the proposed solution.

Tom: Here is my detailed research summary: Comprehensive Research Summary: Probing LLM Veracity with sAwMIL This study critically examines existing methods for assessing the veracity of knowledge encoded within Large Language Models (LLMs) and identifies three fundamental flaws in current literature.

Jane: The central contribution of this work is the introduction of sAwMIL (Sparse-Aware Multiple-Instance Learning), a novel multiclass probing framework designed to address these limitations by modeling LLM veracity along three distinct axes: true, false, and neither.

Title and authors: Tom: Before we get into the details of "The Trilemma of Truth in Large Language Models," let's just quickly look at who wrote this. It’s Germans Savcisens, Tina Eliassi-Rad, and Khoury from Northeastern University and Santa Fe Institute >

Jane: Yeah, I haven't seen their work before. It sounds like a mix of computer science and network science. I wonder what that means for the actual structure of these models >

Tom: It suggests they’re looking at the underlying information flow, not just the surface-level language generation >

Lu: They are focusing on how knowledge is encoded in those internal neural representations after every decoder layer, which is where most of this probing work happens >

Meng: So, if this framework works, it means we can start auditing these models more deeply than just asking them a few questions and looking at the answer >

Lalam: It seems like they are trying to get past the surface-level text and into the actual structure of what's learned >

The paper's summary: Tom: Okay, so what’s actually in this paper? They’re setting up this multiclass probing framework called sAwMIL, which stands for Sparse-Aware Multiple-Instance Learning. It combines multiple-instance learning with conformal prediction >

Jane: That sounds complicated. Can you break down what that means for someone who isn't deep into ML research? What is the main goal here >

Tom: The main goal is to classify a statement as true, false, or neither, based on how the model’s internal representations react to it across different parts of the statement >

Lu: They use a three-stage training process. First they use a Multiple-Instance SVM to get initial signals from bags of token representations >

Meng: So they’re not just looking at one token at the end, but analyzing groups of tokens together, and then refining that signal later >

Lalam: And they use conformal prediction to set statistical guarantees, so it helps them handle cases where the model genuinely doesn't know the answer >

The paper's improvements: Tom: The authors point out three big flaws in older probing methods. They say most of them assume a single axis for veracity, they assume ground-truth labels are perfect proxies, and they assume any signal is localized only at the final token >

Jane: That’s kind of like assuming a car's brake pedal only tells you about stopping, not about acceleration or steering too >

Tom: It is that. The paper suggests sAwMIL fixes that by modeling veracity along three distinct axes: true, false, and neither >

Lu: They introduce causal validation techniques like Contrastive Token Intervention, which lets them see if a specific veracity direction actually influences the model’s output >

Meng: So they aren't just looking for a correlation; they want to find out if this internal signal actually causes the model to produce a different answer >

Lalam: And they use difference-in-differences analysis, which is a fancy way of isolating the veracity effect from general shifts in the model’s overall knowledge base >

Conclusion: Tom: So, to wrap up "The Trilemma of Truth in Large Language Models," they argue that binary truth checking just doesn't cut it for these massive models. They show that common methods fail to give a reliable direction, sometimes performing worse than just prompting the model directly >

Jane: It sounds like the main point is that we need this three-valued logic approach because it captures the complexity of what an LLM actually represents internally >

Tom: Right. sAwMIL consistently outperforms things like TTPD and sPCA probes across different models, showing better generalization and more robust connections to actual model outputs >

Lu: I think the real win here is that they've defined criteria for success—things like correlation on unseen samples, selectivity so it doesn't guess when it should abstain—and achieved those criteria >

Meng: From an engineering standpoint, this means we can build better auditing modules to quantify how reliably a probe identifies a veracity-specific direction across different LLMs >

Lalam: It’s about moving from just checking the output to understanding the internal mechanism that produces that output, and sAwMIL gives us a stronger handle on that mechanism >

Tom: So, "The Trilemma of Truth in Large Language Models" suggests we need a more nuanced way to measure model veracity than simple true or false labels can provide. That’s what we have for this one.

More episodes

← Home