Block Sparse Attention with Log-Linear Complexity
cs.LG
Submitted: 2026-09-25
Updated: 2026-09-25
Code: https://github.com/fla-org/flash-linear-attention
Terminology
Sources
- Just read twice: closing the recall gap for recurrent language models
- IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse
- Longformer: The Long-Document Transformer
- Lessons from the Trenches on Reproducible Evaluation of Language Models
- PIQA: Reasoning about Physical Commonsense in Natural Language
- Generating Long Sequences with Sparse Transformers
- SeerAttention: Learning Intrinsic Sparse Attention in Your LLMs
- RULER: What's the Real Context Size of Your Long-Context Language Models?
- Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling
- DashAttention: Differentiable and Adaptive Sparse Hierarchical Attention
- Simplified Sparse Attention via Gist Tokens
- Pointer Sentinel Mixture Models
- Double-P: Hierarchical Top-P Sparse Attention for Long-Context LLMs
- WinoGrande: An Adversarial Winograd Schema Challenge at Scale
- HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention
- LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexing
- InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptation
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks