Scaling Parameter and Context in Attention: Native Sparse Attention from Mixture-of-Head

arXiv:2609.38832 · cs.CL · Submitted 2026-09-30 · Read on arXiv

cs.CL

Submitted: 2026-09-30

Updated: 2026-09-30

Terminology

• Sparse Attention — 46× in this paper · explained in 5 episodes
• Sparsity — 16× in this paper · explained in 4 episodes
• Attention Mechanism — 8× in this paper · explained in 3 episodes
• KV cache — 6× in this paper · explained in 3 episodes
• Regularization — 5× in this paper · explained in 3 episodes
• Transformer — 5× in this paper · explained in 2 episodes
• Aggregation — 4× in this paper · explained in 1 episode
• Benchmark — 4× in this paper · explained in 2 episodes
• Adam — 2× in this paper · explained in 2 episodes
• Alignment — 2× in this paper · explained in 1 episode
• Embeddings — 2× in this paper · explained in 2 episodes
• Mixture of Block Attention — 2× in this paper
• Accuracy — 1× in this paper · explained in 1 episode
• AdamW Optimizer — 1× in this paper · explained in 1 episode
• Decode — 1× in this paper · explained in 2 episodes
• In-Context Learning — 1× in this paper · explained in 4 episodes
• Mixture-of-Experts (MoE) — 1× in this paper · explained in 7 episodes
• Transformer Models — 1× in this paper · explained in 2 episodes
• Verifier — 1× in this paper · explained in 2 episodes

Sources

• Nemotron-4 340B Technical Report
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
• LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
• Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity
• Evaluating Large Language Models Trained on Code
• BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Training Verifiers to Solve Math Word Problems
• SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
• A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers
• RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse
• Transformer Feed-Forward Layers Are Key-Value Memories
• Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs
• Measuring Massive Multitask Language Understanding
• Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps
• Training Compute-Optimal Large Language Models
• Mixtral of Experts

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering