Faster Than Flash: Exploiting Attention Sparsity for Efficient Long-Context Decoding
cs.LG, cs.AI
Submitted: 2026-08-31
Updated: 2026-08-31
Comments: 20 pages, 8 figures, 10 tables; Accepted at ICML 2026
Code: https://github.com/qluoluo/faster-flash-decoding
License: http://creativecommons.org/licenses/by/4.0/
The gist: The development of long-context Large Language Models (LLMs) is constrained by the memory bandwidth bottleneck and quadratic complexity of the attention mechanism during decoding.
Terminology
Abstract
The development of long-context Large Language Models (LLMs) is constrained by the memory bandwidth bottleneck and quadratic complexity of the attention mechanism during decoding. To overcome the inherent trade-offs between the memory overhead of metadata-based metrics and the computational inefficiency of adaptive selection strategies, we present Faster Flash Decoding (FFD), a novel hardware-algorithm co-design framework designed to break the memory wall in long-context decoding. FFD integrates the selector and computer into a fully fused kernel, replacing external metadata indices with content-aware scanning via low-bit quantization. Furthermore, we introduce the top-delta strategy, which dynamically filters blocks to achieve distribution-adaptive sparsity without global synchronization. Offering a training-free and plug-and-play solution, FFD also enables the reuse of scanning results for computation, achieving up to 11.6x kernel-level speedup and scaling to 256K context length, with 2.37x end-to-end throughput improvement. Empirical validation on RULER and LongBench confirms that FFD maintains model accuracy while delivering high-ratio sparsity, with code available at https://github.com/qluoluo/faster-flash-decoding
Sources
- GPT-4 Technical Report
- Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
- Thus Spake Long-Context Large Language Model
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- Efficient Memory Management for Large Language Model Serving with PagedAttention
- Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
- SparQ Attention: Bandwidth-Efficient LLM Inference
- Twilight: Adaptive Attention Sparsity with Hierarchical Top-$p$ Pruning
- DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models
- Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
- Efficient Streaming Language Models with Attention Sinks
- RULER: What's the Real Context Size of Your Long-Context Language Models?
- FlashDecoding++: Faster Large Language Model Inference on GPUs
- SALE : Low-bit Estimation for Efficient Sparse Attention in Long-context LLM Prefilling
- KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
- No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks