FlashBoB: I/O-Efficient Exact Backward-over-Backward for Softmax Attention
cs.LG, cs.AI
Submitted: 2026-09-21
Updated: 2026-09-21
Code: https://github.com/MingruiLiu-ML-Lab/FlashBoB
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- Longformer: The Long-Document Transformer
- Training Deep Nets with Sublinear Memory Cost
- USP: A Unified Sequence Parallelism Approach for Long Context Generative AI
- GradMem: Learning to Write Context into Memory with Test-Time Gradient Descent
- On Fine-Grained I/O Complexity of Attention Backward Passes
- Online normalizer calculation for softmax
- On First-Order Meta-Learning Algorithms
- RWKV: Reinventing RNNs for the Transformer Era
- Self-attention Does Not Need $O(n^2)$ Memory
- The I/O Complexity of Attention, or How Optimal is Flash Attention?
- FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision
- End-to-End Test-Time Training for Long Context
- Linformer: Self-Attention with Linear Complexity
- FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks