Tracing Computation Density in LLMs
cs.CL, cs.AI, cs.LG
Submitted: 2026-05-26
Updated: 2026-09-09
Comments: Published as a conference paper at EMNLP 2026 (main conference)
License: http://creativecommons.org/licenses/by/4.0/
The gist: Transformer-based large language models (LLMs) are comprised of billions of parameters arranged in deep and wide computational graphs, but it is not clear that they exploit their full capacity for
Terminology
Abstract
Transformer-based large language models (LLMs) are comprised of billions of parameters arranged in deep and wide computational graphs, but it is not clear that they exploit their full capacity for all inputs. We introduce the s-Trace method to efficiently estimate a subgraph of size s that approximates a full model output. With this method, we find the computation in a variety of LLMs to be organized in two distinct phases. A small subgraph mostly composed of early-layer nodes can reconstruct the head of the full model output distribution. Adding further nodes, mostly located in later layers and increasingly consisting of attention heads, leads to incremental refinements in approximating the full output distribution. We find moreover that the amount of necessary computation per input correlates with model uncertainty, and that sparser subgraphs encode shallow statistics, such as unigram frequency. Overall, our results suggest a consistent modular organization in effective LLM computation, with a sparse early-layer core providing a rough prediction that is further refined through denser computations in later layers.
Sources
- Phi-4 Technical Report
- Eliciting Latent Predictions from Transformers with the Tuned Lens
- DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
- Mistral 7B
- Not All Layers of LLMs Are Necessary During Inference
- The Remarkable Robustness of LLMs: Stages of Inference?
- A Primer on the Inner Workings of Transformer-based Language Models
- How Well do LLMs Compress Their Own Chain-of-Thought? A Token Complexity Approach
- The Llama 3 Herd of Models
- 2 OLMo 2 Furious
- Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small
- Qwen3 Technical Report
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering