Mahalanobis-Based Multi-Head Attention for Complex State Propagation
cs.AI
Submitted: 2026-08-25
Updated: 2026-08-25
Code: https://github.com/hilhert/CSP-MHD
Terminology
Sources
- Low-Rank Bottleneck in Multi-head Attention Models
- Combinatoire des Sous-Groupes de Congruence du Groupe Modulaire II
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- Rethinking Attention with Performers
- Attention Mechanisms Through the Lens of Numerical Methods: Approximation Methods and Alternative Formulations
- Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
- DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
- Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces
- Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
- Adam: A Method for Stochastic Optimization
- State Propagation Also Satisfies: A Complex-Valued State-Space Model for Deterministic State Tracking
- Progress measures for grokking via mechanistic interpretability
- Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets
- Retentive Network: A Successor to Transformer for Large Language Models
- Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters
- Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernel
- General Table Question Answering via Answer-Formula Joint Generation
- Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
- VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection