The Topological Trouble With Transformers
cs.LG, cs.AI
Submitted: 2026-04-18
Updated: 2026-09-01
Comments: added citations to recent papers
License: http://creativecommons.org/licenses/by/4.0/
The gist: Transformers encode structure in sequences via an expanding contextual history.
Terminology
Abstract
Transformers encode structure in sequences via an expanding contextual history. However, their purely feedforward architecture fundamentally limits dynamic state tracking. State tracking -- the iterative updating of latent variables reflecting an evolving environment -- involves inherently sequential dependencies that feedforward networks struggle to maintain. Consequently, feedforward models push evolving state representations deeper into their layer stack with each new input step, rendering information inaccessible in shallow layers and ultimately exhausting the model's depth. While this depth limit can be bypassed by dynamic depth models and by explicit or latent thinking that externalizes state representations, these solutions are computationally and memory inefficient. In this article, we argue that temporally extended cognition requires refocusing from explicit thought traces to implicit activation dynamics via recurrent architectures. We introduce a taxonomy of recurrent and continuous-thought transformer architectures, categorizing them by their recurrence axis (depth versus step) and their ratio of input tokens to recurrence steps. Finally, we outline promising research directions, including enhanced state-space models and coarse-grained recurrence, to better integrate state tracking into modern foundation models.
Sources
- Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers
- Physics of Language Models: Part 1, Learning Hierarchical Language Structures
- LLMs Can't Play Hangman: On the Necessity of a Private Working Memory for Language Agents
- Longformer: The Long-Document Transformer
- Modeling Language as a Sequence of Thoughts
- Training-Free Looped Transformers
- Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinking
- Adapting Language Models to Compress Contexts
- Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
- The Collaboration Gap: Exploration and Benchmarking of Open-World Agentic Cooperation
- Addressing Some Limitations of Transformers with Feedback Memory
- Catch Your Breath: Adaptive Computation for Self-Paced Sequence Production
- Better & Faster Large Language Models via Multi-token Prediction
- Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues
- Semantic Tube Prediction: Beating LLM Data Efficiency with JEPA
- Scalable Adaptive Computation for Iterative Generation
- Less is More: Recursive Reasoning with Tiny Networks
- Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughts
- LLMs Get Lost In Multi-Turn Conversation
- Skip a Layer or Loop it? Test-Time Depth Adaptation of Pretrained LLMs
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks