Tracing the complexity profiles of different linguistic phenomena through the intrinsic dimension of LLM representations
cs.CL
Submitted: 2026-01-07
Updated: 2026-08-28
Comments: Published as a conference paper at EMNLP 2026
Journal ref: Proceedings of EMNLP 2026
Code: https://github.com/franfranz/syntactic_complexity_in_LLMs
License: http://creativecommons.org/licenses/by/4.0/
The gist: We explore intrinsic dimension (ID) of LLM representations as a marker of linguistic complexity.
Terminology
Abstract
We explore intrinsic dimension (ID) of LLM representations as a marker of linguistic complexity. Specifically, we test whether ID differences across model layers reflect well-known complexity contrasts established in (psycho)linguistics: coordination vs. subordination, right-branching vs. center-embedding, and unambiguous vs. ambiguous attachment. Our results on six different LLMs show that these contrasts are consistently reflected in ID differences, with more complex phenomena eliciting higher ID profiles. Notably, ID differences emerge at different points across layers for different contrasts, also reaching their peaks at different stages. Further experiments using representational similarity and layer pruning confirm the trends. We conclude that ID is a useful marker of linguistic complexity in LLMs, that it points to similar linguistic processing steps across disparate LLMs, and that it has the potential to differentiate between different types of complexity.
Sources
- Mechanisms vs. Outcomes: Probing for Syntax Fails to Explain Performance on Targeted Syntactic Evaluations
- A quantitative analysis of semantic information in deep representations of text and images
- The Llama 3 Herd of Models
- A Primer on the Inner Workings of Transformer-based Language Models
- Mistral 7B
- How Linguistics Learned to Stop Worrying and Love the Language Models
- The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models
- Model Internal Sleuthing: Finding Lexical Identity and Inflectional Features in Modern Language Models
- Qwen2.5 Technical Report
- Gemma 2: Improving Open Language Models at a Practical Size
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering