A Comparative Analysis of Attention versus State-Space Models for In-Context Learning
cs.LG
Submitted: 2026-09-26
Updated: 2026-09-26
Terminology
Sources
- Transformers learn to implement preconditioned gradient descent for in-context learning
- What learning algorithm is in-context learning? Investigations with linear models
- Zoology: Measuring and Improving Recall in Efficient Language Models
- Simple linear attention language models balance the recall-throughput tradeoff
- Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection
- xLSTM: Extended Long Short-Term Memory
- Birth of a Transformer: A Memory Viewpoint
- From Markov to Laplace: How Mamba In-Context Learns Markov Chains
- Convergence of Gradient Descent for Recurrent Neural Networks: A Nonasymptotic Analysis
- Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality
- Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers
- On the Expressivity of Selective State-Space Layers: A Multivariate Polynomial Approach
- In-Context Learning of Linear Dynamical Systems with Transformers: Approximation Bounds and Depth-Separation
- Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
- Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models
- The Evolution of Statistical Induction Heads: In-Context Learning Markov Chains
- What One Cannot, Two Can: Two-Layer Transformers Provably Represent Induction Heads on Any-Order Markov Chains
- Hungry Hungry Hippos: Towards Language Modeling with State Space Models
- What Can Transformers Learn In-Context? A Case Study of Simple Function Classes
- Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks