Switch Attention: Towards Dynamic and Fine-grained Hybrid Transformers

arXiv:2603.26380 · cs.CL · Submitted 2026-03-27 · Read on arXiv

cs.CL

Submitted: 2026-03-27

Updated: 2026-08-29

Terminology

• Transformer — 84× in this paper · explained in 2 episodes
• Regularization — 22× in this paper · explained in 3 episodes
• Accuracy — 16× in this paper · explained in 1 episode
• Attention Mechanism — 11× in this paper · explained in 2 episodes
• Benchmark — 10× in this paper · explained in 2 episodes
• KV cache — 8× in this paper · explained in 3 episodes
• Sparse Attention — 6× in this paper · explained in 5 episodes
• Decode — 5× in this paper · explained in 2 episodes
• Transformer Architecture — 5× in this paper · explained in 2 episodes
• Transformer Models — 5× in this paper · explained in 2 episodes
• Hybrid Models — 4× in this paper · explained in 2 episodes
• Adam — 3× in this paper · explained in 2 episodes
• Perplexity — 3× in this paper · explained in 3 episodes
• Ablation — 2× in this paper · explained in 1 episode
• Ablation Studies — 2× in this paper · explained in 1 episode
• Aggregation — 2× in this paper · explained in 1 episode
• Core — 2× in this paper · explained in 2 episodes
• Natural Questions — 2× in this paper
• Quantization — 2× in this paper · explained in 12 episodes
• Sparsity — 2× in this paper · explained in 4 episodes
• Alignment — 1× in this paper · explained in 1 episode
• Embeddings — 1× in this paper · explained in 2 episodes
• Natural Language Inference (NLI) — 1× in this paper · explained in 2 episodes
• Speculative Decoding — 1× in this paper · explained in 2 episodes

Sources

• Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
• Composer: A Search Framework for Hybrid Neural Architecture Design
• Generating Long Sequences with Sparse Transformers
• Gemini: A Family of Highly Capable Multimodal Models
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Just read twice: closing the recall gap for recurrent language models
• Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes
• Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
• Longformer: The Long-Document Transformer
• Native Hybrid Attention for Efficient Sequence Modeling
• Better & Faster Large Language Models via Multi-token Prediction
• Jet-Nemotron: Efficient Language Model with Post Neural Architecture Search
• Log-Linear Attention
• R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routing
• Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengths
• RULER: What's the Real Context Size of Your Long-Context Language Models?
• HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing
• Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
• Gemma 2: Improving Open Language Models at a Practical Size
• Mistral 7B

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering