Self-Aware Knowledge Probing: Evaluating Language Models' Relational Knowledge through Confidence Calibration
summary
The gist
The gist The proposed framework assesses model reliability by probing relational knowledge through three confidence estimation modalities: intrinsic confidence, structural consistency, and semantic
In short
This research developed a framework to assess how well language models know facts by looking at their confidence scores. The study found that most models are overconfident, meaning they think they are more certain than they actually are. The best way to measure reliable knowledge is through structural consistency estimates and accounting for how statements can be rephrased.
Key concepts
- Knowledge Probing
- This is a method used to check how much relational knowledge a language model has learned during its training. It acts like a diagnostic tool, quantifying the specific facts or relationships the model has internalized. The goal is to measure the depth and reliability of this acquired knowledge.
- Confidence Calibration
- Calibration refers to how accurately a model's confidence scores reflect its true level of certainty regarding an answer. If a model is well-calibrated, when it says it is 80% sure, it should be correct 80% of the time. Poor calibration means the model is often overconfident or underconfident.
- Intrinsic Confidence
- This measure comes from the raw mathematical outputs of a language model's prediction process, specifically its logprobabilities. It includes metrics like Base-Confidence and Margin-Confidence, which look at how spread out the probabilities are. These scores are used to estimate a model's internal certainty before considering external factors.
- Structural Consistency
- This modality evaluates confidence by checking if the model's answer remains consistent even when the question or statement is slightly rephrased. It tests whether the model understands the underlying relationship rather than just memorizing a specific phrasing, making it a robust measure of relational knowledge.
Terminology used across episodes
This episode discusses
- Self-Aware Knowledge Probing: Evaluating Language Models' Relational Knowledge through Confidence Calibration · Paper Radio
- Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models
- On Calibration of Modern Neural Networks
- A Better Way to Do Masked Language Model Scoring
- Can multiple-choice questions really be useful in detecting the abilities of LLMs?
- Teaching Models to Express Their Uncertainty in Words
- Measuring Calibration in Deep Learning
- Language Models as Knowledge Bases?
- Identifying Mislabeled Data using the Area Under the Margin Ranking
- LM-PUB-QUIZ: A Comprehensive Framework for Zero-Shot Evaluation of Relational Knowledge in Language Models
- Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback
- BEAR: A Unified Framework for Evaluating Relational Knowledge in Causal and Masked Language Models
- Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs
- How Reliable are LLMs as Knowledge Bases? Re-thinking Facutality and Consistency
- Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Models
The paper
Self-Aware Knowledge Probing: Evaluating Language Models' Relational Knowledge through Confidence Calibration · Read on arXiv
Christopher Kissling, Elena Merdjanovska, Alan Akbik
Humboldt-Universität zu Berlin
Knowledge probing quantifies how much relational knowledge a language model (LM) has acquired during pre-training. Existing knowledge probes evaluate model capabilities through metrics like prediction accuracy and precision. Such evaluations fail to account for the model's reliability, reflected in the calibration of its confidence scores. In this paper, we establish an empirical framework to evaluate the calibration of relational knowledge in a closed answer-set setting. We cover three modalities of model confidence: (1) intrinsic confidence, (2) paraphrase consistency and (3) linguistic confidence. Our extensive analysis of 21 causal and 8 masked language models reveals that most models, but especially those pre-trained with the masking objective, are overconfident. The best-calibrated scores come from confidence estimates that account for inconsistencies due to statement rephrasing. Moreover, calibration improves when increasing the number of answer options.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.
Jane: Today's paper: "Self-Aware Knowledge Probing".
Tom: The gist The proposed framework assesses model reliability by probing relational knowledge through three confidence estimation modalities: intrinsic confidence, structural consistency, and semantic grounding.
Jane: First, who's behind it and why it matters.
Paper summary: Tom: So, we've talked about the basics of this paper, "Self-Aware Knowledge Probing: Evaluating Language Models' Relational Knowledge through Confidence Calibration." We’ve established that existing methods often ignore how reliable a model’s confidence scores actually are >
Jane: That’s right. This paper proposes a novel calibration probing framework for relational knowledge that covers three modalities of model confidence: intrinsic confidence, structural consistency, and semantic grounding >
Lu: The main thrust is to move beyond just measuring attainment—which is whether the model can predict the correct token—and focus on the model’s awareness of its own knowledge boundaries >
Meng: They use a unified scoring rule called BEAR which shifts things from open-ended generation to a closed-set ranking by estimating the full log-likelihood for each statement created with a template >
Lalam: The results show that most models, especially those pre-trained with the masking objective, are overconfident when measured by these different confidence estimates >
Tom: They specifically point out that "CMargin" consistently performs better than "CBase" because CBase is markedly overconfident >
Jane: And they found a reliability gap where MLMs remain fundamentally less aware of their own knowledge boundaries than those from causal language models, despite similar factual accuracy >
Lu: Also, they noticed a scale-dependency for semantic grounding: smaller models seem "semantically blind" to expressions of doubt, but larger models start aligning their internal confidence with linguistic cues >
Meng: That’s a practical concern. If smaller models don't register uncertainty well, it limits how much we can trust them when they are unsure about something >
Lalam: They also looked at adding epistemic markers like "certainly" or "possibly," and those markers mostly hurt calibration because LMs are consistently overconfident >
Tom: So, the key thing here is that getting well-calibrated confidences for relational knowledge is really hard, which hints that LMs might not have a deeper understanding of their own knowledge boundaries >
Jane: Exactly. The paper shows that regardless of the pre-training objective or the model family you pick, confidences tend to be overconfident, and the MLM scoring generally makes this overconfidence behavior worse >
Lu: It seems like they’re pushing for a more self-aware way to probe knowledge where the AI can assess its own uncertainty >
Meng: From an engineering standpoint, if we can get better at predicting these calibrated scores, it helps us build systems that are more transparent about when they might be wrong >
Lalam: It gives us a way to assess the quality of the relational knowledge internalized during pre-training, not just whether a single answer is correct >
Conclusion: Tom: So wrapping up this discussion on "Self-Aware Knowledge Probing: Evaluating Language Models' Relational Knowledge through Confidence Calibration." The authors are looking at how much relational knowledge language models have learned during pre-training and how reliable their confidence scores are >
Jane: They propose a framework that looks at intrinsic confidence, structural consistency, and semantic grounding to get a more nuanced view of model reliability >
Lu: The main implication is that current methods for measuring knowledge probes focus too much on attainment, like just predicting the correct token, instead of assessing the model’s awareness of its own knowledge boundaries >
Meng: This suggests we need better ways to evaluate AI systems that can tell us when they are unsure about a fact or a relationship >
Lalam: The paper shows that even with different training objectives, like masking versus causal language models, there's a consistent tendency for overconfidence in the scores >
Tom: And it points toward the idea that we need self-aware knowledge probing to truly understand how these models know what they know and where their limits are >
More episodes
- 2405.01930-OARelatedWork: A Large-Scale Dataset of Related Work Sections with Full-texts from Open Access Sources
- 2412.12478-Human-in-the-Loop Generation of Adversarial Texts: A Case Study on Tibetan Script
- 2502.01025-Knowing When to Stop: Efficient Context Processing via Latent Sufficiency Signals
- 2504.02891-Automated Survey Collection with LLM-based Conversational Agents
- 2505.17244-ReasoningShield: Safety Detection over Reasoning Traces of Large Reasoning Models
- 2505.22184-Breaking the Cloak! Unveiling Chinese Cloaked Toxicity with Homophone Graph and Toxic Lexicon
- 2510.10998-ABLEIST: Intersectional Disability Bias in LLM-Generated Hiring Scenarios
- 2510.13154-The Alignment Veto: How Safety Training Suppresses Cultural Knowledge in LLMs
- 2601.17898-Assessment of Generative Named Entity Recognition in the Era of Large Language Models
- 2602.01709-ARTIS: Agentic Risk-Aware Test-Time Scaling via Iterative Simulation