The Evolution of Attention in Large Language Models: Mechanisms, Trade-offs, and Emerging Trends

arXiv:2609.39661 · cs.CL · Submitted 2026-09-30 · Read on arXiv

cs.CL

Submitted: 2026-09-30

Updated: 2026-09-30

Code: https://github.com/meta-llama/llama-models

Terminology

• Sparse Attention — 86× in this paper · explained in 5 episodes
• Transformer — 72× in this paper · explained in 2 episodes
• Sparsity — 17× in this paper · explained in 4 episodes
• Core — 15× in this paper · explained in 2 episodes
• Distillation — 14× in this paper · explained in 2 episodes
• Addressability — 12× in this paper · explained in 1 episode
• Aggregation — 12× in this paper · explained in 1 episode
• Hybrid Architectures — 12× in this paper · explained in 3 episodes
• Adaptation — 11× in this paper · explained in 1 episode
• Alignment — 11× in this paper · explained in 1 episode
• KV cache — 10× in this paper · explained in 3 episodes
• Associative Memory — 7× in this paper · explained in 2 episodes
• Decoupling — 7× in this paper · explained in 2 episodes
• Redundancy — 7× in this paper · explained in 2 episodes
• Adaptivity — 6× in this paper · explained in 1 episode
• Attention Mechanism — 4× in this paper · explained in 3 episodes
• Fidelity — 4× in this paper · explained in 2 episodes
• International Conference on Learning Representations — 4× in this paper
• Benchmark — 3× in this paper · explained in 2 episodes
• Decode — 3× in this paper · explained in 2 episodes
• Dynamic Linear Attention — 3× in this paper
• MiniMax Sparse Attention — 3× in this paper
• Multi-Head Linear Attention — 3× in this paper
• Qwen Sparse Attention — 3× in this paper
• Sparse Delta Memory — 3× in this paper
• Sparse State Expansion — 3× in this paper
• Trace — 3× in this paper · explained in 2 episodes
• Access Regimes — 2× in this paper · explained in 1 episode
• Accuracy — 2× in this paper · explained in 1 episode
• Benchmarking — 2× in this paper · explained in 3 episodes
• Clustering — 2× in this paper · explained in 2 episodes
• Compressed Sparse Attention — 2× in this paper
• Consistency — 2× in this paper · explained in 3 episodes
• DeepSeek Sparse Attention — 2× in this paper
• Fine-Tuning — 2× in this paper · explained in 5 episodes
• LongCat Sparse Attention — 2× in this paper
• Native Sparse Attention — 2× in this paper
• Recurrent Memory Transformer — 2× in this paper
• Regularization — 2× in this paper · explained in 3 episodes
• Reinforcement Learning — 2× in this paper · explained in 4 episodes
• Softmax Linear Attention — 2× in this paper
• Structured State Space Duality — 2× in this paper
• Transformer Models — 2× in this paper · explained in 2 episodes
• Verification — 2× in this paper · explained in 2 episodes
• Calibration — 1× in this paper · explained in 7 episodes
• Cosine Similarity — 1× in this paper · explained in 3 episodes
• Credit assignment — 1× in this paper · explained in 3 episodes
• Cross-Layer Value Routing — 1× in this paper
• Embeddings — 1× in this paper · explained in 2 episodes
• Foundation Model — 1× in this paper · explained in 3 episodes
• Foundation Models — 1× in this paper · explained in 7 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• Heavily Compressed Attention — 1× in this paper
• Hybrid Models — 1× in this paper · explained in 2 episodes
• In-Context Learning — 1× in this paper · explained in 4 episodes
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
• Latent Representation — 1× in this paper · explained in 2 episodes
• Mixture of Attention Heads — 1× in this paper
• Quantization — 1× in this paper · explained in 12 episodes
• State Management — 1× in this paper · explained in 3 episodes
• State Space Models (SSMs) — 1× in this paper · explained in 4 episodes
• State-Memory Attention — 1× in this paper
• Structural Constraints — 1× in this paper · explained in 2 episodes
• Transformer Architecture — 1× in this paper · explained in 2 episodes

Sources

• Attention Is All You Need
• Language Models are Few-Shot Learners
• LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
• RULER: What's the Real Context Size of Your Long-Context Language Models?
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
• Kwai Summary Attention Technical Report
• Fast Transformer Decoding: One Write-Head is All You Need
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
• Reducing Transformer Key-Value Cache Size with Cross-Layer Attention
• Longformer: The Long-Document Transformer
• Big Bird: Transformers for Longer Sequences
• Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
• Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling
• IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse
• LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexing
• Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
• Retentive Network: A Successor to Transformer for Large Language Models
• Linear Transformers Are Secretly Fast Weight Programmers
• Gated Delta Networks: Improving Mamba2 with Delta Rule

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering