What Language Models Know But Don't Say: Non-Generative Prior Extraction for Generalization

summary

The gist

The following is a detailed summary of the scientific paper: Problem and Motivation In domains such as medicine and finance, "large-scale labeled data is costly and often unavailable," leading to

In short

The episode discusses the paper 'What Language Models Know But Don't Say,' which proposes using LLMs not as text generators but as structured knowledge bases. The method, LoID, extracts internal structural data from model logits to create reliable statistical priors. This approach significantly improves predictive modeling when dealing with out-of-distribution data.

Key concepts

Logits: Logits are the raw output scores of a Language Model before they are converted into probabilities or text. Instead of prompting the LLM for a direct answer, researchers analyze these underlying scores to determine if the model favors a positive or negative effect regarding a specific feature.
LoID (Logit-Informed Distributions): This is the core method where structural knowledge is extracted from LLM logits. It uses carefully constructed semantic pairings to measure an LLM's internal belief about a feature's influence, allowing researchers to quantify the strength and reliability of that knowledge.
Non-Generative Extraction: A technique where AI is used not to predict or generate text, but its internal structural knowledge is extracted instead. This method is deterministic and repeatable, providing reliable constraints for statistical models rather than relying on variable output.
Out-of-Distribution (OD) Data: This refers to data that differs significantly from the data a model was originally trained on. The paper demonstrates that using the LoID method helps AI models perform much better when encountering this real-world shift or uncertainty.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Federated Learning — explained in 16 episodes
Retrieval-Augmented Generation (RAG) — explained in 16 episodes
Interpretability — explained in 15 episodes
Sample Complexity — explained in 15 episodes
Generalization — explained in 13 episodes

This episode discusses

The paper

What Language Models Know But Don't Say: Non-Generative Prior Extraction for Generalization · Read on arXiv

Mohammad Ghassemi, Sara Rezaeimanesh, Michigan State University

In domains like medicine and finance, large-scale labeled data is costly and often unavailable, leading to models trained on small datasets that struggle to generalize to real-world populations. Large language models contain extensive knowledge from years of research across these domains. We propose LoID (Logit-Informed Distributions), a deterministic method for extracting informative prior distributions for Bayesian logistic regression by directly accessing their token-level predictions. Rather than relying on generated text, we probe the model's confidence in opposing semantic directions (positive vs. negative impact) through carefully constructed sentences. By measuring how consistently the LLM favors one direction across diverse phrasings, we extract the strength and reliability of the model's belief about each feature's influence. We evaluate LoID on ten real-world tabular datasets under synthetic out-of-distribution (OOD) settings characterized by covariate shift, where the training data represents only a subset of the population. We compare our approach against (1) standard uninformative priors, (2) AutoElicit, a recent method that prompts LLMs to generate priors via text completions, (3) LLMProcesses, a method that uses LLMs to generate numerical predictions through in-context learning and (4) an oracle-style upper bound derived from fitting logistic regression on the full dataset. We assess performance using Area Under the Curve (AUC). Across datasets, LoID significantly improves performance over logistic regression trained on OOD data, recovering up to 59% of the performance gap relative to the oracle model. LoID outperforms AutoElicit and LLMProcessesc on 8 out of 10 datasets, while providing a reproducible and computationally efficient mechanism for integrating LLM knowledge into Bayesian inference.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "What Language Models Know But Don't Say: Non-Generative Prior Extraction for Generalization".

Jane: The paper was written by Mohammad Ghassemi, Sara Rezaeimanesh and Michigan State University from.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Jane: We also have Lu with us today — senior AI researcher at Tsinghua.

Tom: We also have Meng with us today — lead engineer at a mysterious AI startup.

Jane: We also have Lalam with us today — the in-house Large Language Model.

Tom: Alright, let's get started.

Title: Tom: So, we’re looking at the title again, "What LLMs Know but Don’t Say," which hints at this internal knowledge base. The authors are proposing a way to use this knowledge without relying on the typical outputs of AI models—that's what "Non-generative" means.

Jane: It's a huge step away from prompting an LLM to just predict an answer; instead, they are extracting structural information from the logits. This is crucial because generating text is inherently variable, so unreliable for precise scientific modeling.

Lu: The implication here is that we can use AI not just as a source of answers, but as a source of reliable constraints or guiding rules for Bayesian statistical models. It’s about using the structure of knowledge rather than the content.

Meng: If we are only looking at the logits, it means this method is deterministic and repeatable, which is exactly what any real-world engineer needs to deploy a model reliably in a high-stakes environment.

Lalam: A deterministic way to know what’s true about a feature's impact—that’s huge for confidence. It moves us away from the "hallucination" problem and toward using AI as a structured knowledge base.

Summary of Paper's Claims: Tom: The paper summarizes how LoID, or Logit-Informed Distributions, works by probing the LLM with specific semantic pairings for each feature. It’s not just asking "what is the impact," but using carefully constructed sentences to see if the model favors a positive or negative effect.

Jane: It measures that preference by looking at how often the LLM predicts specific tokens, like "positive" or "negative." This helps quantify exactly how much of a belief an LLM has about a feature’s influence.

Lu: And the paper claims that this quantification—the strength and reliability of the belief—is what forms Normal priors for the coefficients in regression models. It’s turning raw LLM probability into statistical parameters.

Meng: The core claim is that these generated priors are much more informative than just using standard, uninformative Bayesian priors, especially when we are dealing with out-of-distribution data. That's the central hypothesis we need to test practically.

Lalam: It suggests that the LLM’s internal knowledge about how things usually work—like how age influences income—can be systematically extracted and applied to guide our statistical models toward better outcomes.

Improvements Suggested by the Paper: Tom: So, we've seen how it works; now let's talk about the results. The paper suggests that LoID significantly improves performance when trained on out-of-distribution OOD data. This is a critical improvement in generalizing beyond the training set.

Jane: They found that LoID often outperforms existing methods, like LLMProcesses and AutoElicit, across fifteen different datasets, which are benchmarks in healthcare and finance. The paper shows consistent performance gains compared to the original OOD model.

Lu: The best part of the improvement is that it recovers up to fifty percent of the performance gap relative to a perfect oracle model. That means we’re getting much closer to peak performance than we thought possible with current methods.

Meng: I found that especially impressive; achieving that much of the oracle's potential without actually having access to the full, true distribution is a massive win for practical deployment under uncertainty.

Lalam: It implies that by incorporating this external domain knowledge, we are making our AI models far more robust and dependable when they will inevitably face real-world data shift.

Conclusion: Tom: We’ve covered a lot of ground today, from the technical details of LoID to its impressive results on out-of-distribution datasets. It really shows that LLMs are much more than just chat bots; they contain structured, useful knowledge.

Jane: I think the major takeaway is that we can use this internal AI knowledge base—the "what LLMs know but don't say"—to build much more reliable statistical models for real-world applications.

Lu: It’s a paradigm shift, turning latent knowledge into actionable data parameters, and it opens up huge possibilities for the future of predictive modeling.

Meng: My final thought is that this method is efficient, which is vital for practical implementation in any complex system design.

Lalam: To summarize our discussion on "What LLMs Know but Don’t Say: Non-generative Prior Extraction for Generalization," we see a future where AI acts as an intelligent guide to improve the reliability of our most critical predictive models.

More episodes

← Home