The Changing Geometry of Grammar: Dimensionality and Neighborhood Reorganization across Transformer Layers

arXiv:2608.25166 · cs.CL · Submitted 2026-08-25 · Read on arXiv

cs.CL

Submitted: 2026-08-25

Updated: 2026-08-25

Terminology

• Accuracy — 16× in this paper · explained in 1 episode
• Intrinsic Dimension — 15× in this paper · explained in 2 episodes
• Transformer — 7× in this paper · explained in 2 episodes
• AUC — 6× in this paper · explained in 2 episodes
• Information Imbalance — 6× in this paper
• Abstraction — 4× in this paper · explained in 1 episode
• Core — 3× in this paper · explained in 2 episodes
• Trace — 3× in this paper · explained in 2 episodes
• Alignment — 2× in this paper · explained in 1 episode
• Ablation — 1× in this paper · explained in 1 episode
• Ablation Studies — 1× in this paper · explained in 1 episode
• Adaptive Binomial Intrinsic Dimension Estimator — 1× in this paper
• Embeddings — 1× in this paper · explained in 2 episodes
• F1 Score — 1× in this paper · explained in 3 episodes
• Fisher Information — 1× in this paper · explained in 3 episodes
• Homogeneity — 1× in this paper · explained in 2 episodes
• Intrinsic Dimensionality — 1× in this paper
• Tokenization — 1× in this paper · explained in 2 episodes
• Transformer Models — 1× in this paper · explained in 2 episodes

Sources

• Scale adaptive and robust intrinsic dimension estimation via optimal neighbourhood identification
• The Pile: An 800GB Dataset of Diverse Text for Language Modeling
• Disentangling Geometry, Performance, and Training in Language Models
• The Geometry of Tokens in Internal Representations of Large Language Models
• Probing Geometry of Next Token Prediction Using Cumulant Expansion of the Softmax Entropy

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering