Large Language Models Develop Belief State Geometry In-Context
cs.LG, cs.CL
Submitted: 2026-09-15
Updated: 2026-09-15
Comments: 87 pages
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- Eliciting Latent Predictions from Transformers with the Tuned Lens
- Stories in Space: In-Context Learning Trajectories in Conceptual Belief Space
- Extracting Algorithms in Pre-trained LLMs: A Case on Hidden Markov Models
- Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds
- The Llama 3 Herd of Models
- A Theory of Emergent In-Context Learning as Implicit Structure Induction
- Taxonomy of Prediction
- The Hydra Effect: Emergent Self-repair in Language Model Computations
- The Dead Salmons of AI Interpretability
- Next-token pretraining implies in-context learning
- Neural networks leverage nominally quantum and post-quantum representations
- The Shape of Beliefs: Geometry, Dynamics, and Interventions along Representation Manifolds of Language Models' Posteriors
- Steering Language Models With Activation Engineering
- Larger language models do in-context learning differently
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks