Clustered Attractor Manifolds and Dynamical Condensation in Self-Attention

arXiv:2608.08922 · cond-mat.dis-nn, cond-mat.stat-mech, cs.LG · Submitted 2026-08-09 · Read on arXiv

Qucheng Gao, Zuyi Yang, Xiao Chen

cond-mat.dis-nn, cond-mat.stat-mech, cs.LG

Submitted: 2026-08-09

Updated: 2026-08-11

Comments: 6 pages, 3 figures. Supplemental Material: 25 pages, 9 figures

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Transformer — 24× in this paper · explained in 2 episodes
• Clustering — 9× in this paper · explained in 2 episodes
• Benchmark — 6× in this paper · explained in 2 episodes
• Alignment — 5× in this paper · explained in 1 episode
• Dynamical regimes — 2× in this paper · explained in 2 episodes
• Entropy Collapse — 2× in this paper · explained in 2 episodes
• Mean-Field Dynamics — 2× in this paper · explained in 2 episodes
• Metastability — 2× in this paper · explained in 2 episodes
• Robustness — 2× in this paper · explained in 5 episodes
• Transformer Architecture — 2× in this paper · explained in 2 episodes
• Transformer Models — 2× in this paper · explained in 2 episodes
• Accuracy — 1× in this paper · explained in 1 episode
• Attention Entropy — 1× in this paper · explained in 2 episodes
• Failure Modes — 1× in this paper · explained in 3 episodes
• Multimodal Learning — 1× in this paper · explained in 2 episodes

Sources

• Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View
• The emergence of clusters in self-attention dynamics
• A mathematical perspective on Transformers
• Analysis of mean-field models arising from self-attention dynamics in transformer architectures with layer normalization
• Multistability of Self-Attention Dynamics in Transformers
• PureNav: A Personalized Navigation Service for Environmental Justice Communities Impacted by Planned Disruptions
• Robust Probabilistic Bisimilarity for Labelled Markov Chains
• A multiscale analysis of mean-field transformers in the moderate interaction regime
• Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisation
• Dynamic metastability in the self-attention model
• Rethinking Attention with Performers

Related papers

• Few-Shot Neuromorphic Vision in a Nonlinear Photonic Network Laser
• Hyperbolic lattices with mass disorder: Phases and phase transitions
• Adaptive Neural Quantum States: A Recurrent Neural Network Perspective
• Machine learning Majorana topology using unsupervised and supervised learning
• Quenched fluctuation-induced force arising from polarization disorder