Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruning

arXiv:2601.06787 · cs.CL · Submitted 2026-01-11 · Read on arXiv

cs.CL

Submitted: 2026-01-11

Updated: 2026-10-03

Comments: Accepted to EMNLP 2026 (Main)

Code: https://github.com/EleutherAI/lm-evaluation-harnesshttps:

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Pruning — 76× in this paper · explained in 4 episodes
• Redundancy — 35× in this paper · explained in 2 episodes
• Accuracy — 31× in this paper · explained in 1 episode
• Transformer — 13× in this paper · explained in 2 episodes
• Ablation — 12× in this paper · explained in 1 episode
• Robustness — 6× in this paper · explained in 5 episodes
• Benchmark — 5× in this paper · explained in 2 episodes
• Perplexity — 5× in this paper · explained in 3 episodes
• Attention Mechanism — 4× in this paper · explained in 2 episodes
• Block Influence — 3× in this paper
• Large Language Models (LLMs) — 3× in this paper · explained in 6 episodes
• Positional Bias — 3× in this paper · explained in 2 episodes
• Core — 2× in this paper · explained in 2 episodes
• Sparsity — 2× in this paper · explained in 4 episodes
• Ablation Studies — 1× in this paper · explained in 1 episode
• Clustering — 1× in this paper · explained in 2 episodes
• Consistency — 1× in this paper · explained in 3 episodes
• Cosine Similarity — 1× in this paper · explained in 3 episodes
• Downstream Tasks — 1× in this paper · explained in 2 episodes
• Elias — 1× in this paper · explained in 2 episodes
• Feature Extraction — 1× in this paper · explained in 2 episodes
• Grouped-Query Attention — 1× in this paper
• Sparse Attention — 1× in this paper · explained in 5 episodes
• Structural Integrity — 1× in this paper · explained in 3 episodes
• Transformer Architecture — 1× in this paper · explained in 2 episodes
• Verification — 1× in this paper · explained in 2 episodes
• World Model — 1× in this paper · explained in 7 episodes

Sources

• The Llama 3 Herd of Models
• What Matters in Transformers? Not All Attention is Needed
• The Universal Weight Subspace Hypothesis
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• High-Layer Attention Pruning with Rescaling
• Qwen3 Technical Report
• Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coin
• Entropy-Based Block Pruning for Efficient Large Language Models
• WinoGrande: An Adversarial Winograd Schema Challenge at Scale
• Gemma 3 Technical Report
• FinerCut: Finer-grained Interpretable Layer Pruning for Large Language Models
• A Survey of Large Language Models

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering