When Does Recurrence Become an Algorithm? Convergence Selection in Weight-Tied Looped Transformers
Tong Zhang, Junhao Hu, Yun Peng, Tao Xie
cs.LG, cs.AI, stat.ML
Submitted: 2026-07-22
License: http://creativecommons.org/licenses/by/4.0/
The gist: When does a weight-tied looped transformer -- one block applied T times -- implement an actual algorithm? We answer with four findings from controlled populations on group word problems.
Terminology
Abstract
When does a weight-tied looped transformer -- one block applied T times -- implement an actual algorithm? We answer with four findings from controlled populations on group word problems. (1) The budget law: free training installs a linear computation frontier, a mechanism that solves v positions per loop, whose speed is priced by the training contract: v n train/T train (exponent 0.98 +/- 0.04, R 2=0.99), exactly unity under T=n training. SGD selects a frontier matching the minimum the contract demands; granting more test-time loops than ever trained rescues late positions at fixed input length, yielding a principled halting rule T* = ceil(n / v-hat). (2) Architecture prior, not expressivity, picks the algorithm: standard-depth transformers learn parallel scans on this family; weight tying flips the selection to the serial frontier, even when positional addressing for a log-depth scan is supplied. At matched depth and parameters, untied models extrapolate worst and fail to learn A5 at all. (3) The walls are not where circuit complexity says: NC1-completeness costs nothing (A5 generalizes fully), while group order does (S5's 120x120 operator deadlocks joint learning) -- and an operator-first curriculum dissolves the wall in every seed. (4) Mechanisms are portable, not mandatable: warm-starting across budget contracts transfers the algorithm in every seed, re-pricing its speed, while imposing seriality through the input schedule fails where free training succeeds. These results are invisible to standard instruments, which provably saturate at the fixed points trained loops converge to. We introduce a head instrument, the convergence-time scaling tau(n,i), validate it causally via damage cones whose slope reproduces v, and show in-distribution head measurements predict out-of-distribution fate where tail metrics do not. Results replicate on the public easy-to-hard benchmark.
Sources
- A Mechanistic Analysis of Looped Reasoning Language Models
- Scaling Latent Reasoning via Looped Language Models
- DifFRACT: Diffusion Feature Reconstruction and Attribution for Circuit Tracing
- Looped Transformers for Length Generalization
- Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
- Why are Sensitive Functions Hard for Transformers?
- Step-resolved data attribution for looped transformers
- Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers
- Latent Chain-of-Thought? Decoding the Depth-Recurrent Transformer
- A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers
- The Illusion of State in State-Space Models
- Looped Transformers are Better at Learning Learning Algorithms
- Enhancing Auto-regressive Chain-of-Thought through Loop-Aligned Reasoning
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks