Beyond Depth Truncation: Controlled Evaluation of Depth Utilization in Recursive Language Models
cs.AI
Submitted: 2026-09-17
Updated: 2026-09-17
License: http://creativecommons.org/licenses/by/4.0/
The gist: Depth-recurrent language models iteratively apply a small layer stack, decoupling per-token compute from distinct parameter count.
Terminology
Abstract
Depth-recurrent language models iteratively apply a small layer stack, decoupling per-token compute from distinct parameter count. To determine whether such a model genuinely utilizes its depth, both recurrence and layer-pruning literatures rely on a shared evaluation: truncating depth at inference time, plotting quality against retained depth fraction, and reading off the slope. While cheap and training-free, this metric suffers from an unexamined flaw: it extracts a single scalar from an intervention that alters multiple model properties simultaneously. Depth truncation concurrently reduces the number of block applications, decreases the volume of distinct computation performed, and pushes the readout head onto an out-of-distribution residual stream. The observed slope conflates all three factors, yet is conventionally interpreted as reflecting solely the second. We propose the Depth Control Protocol (DCP), a diagnostic suite that disentangles these three quantities. DCP comprises three positive controls that isolate each factor while varying the others, a negative control applying the identical interventions to dense transformers to ensure the effect is not an artifact of the measurement protocol, and a controlled training intervention to verify causality. The linchpin control, running the full budget of block applications while executing only a single distinct iteration, is strictly realizable only in depth-wise weight-sharing architectures, since in a dense network repeating a layer yields an entirely different model rather than the same model in an alternative configuration.
Sources
- PonderNet: Learning to Ponder
- Universal Transformers
- Depth-Adaptive Transformer
- Reducing Transformer Depth on Demand with Structured Dropout
- SCORE: Replacing Layer Stacking with Contractive Recurrent Depth
- Adaptive Computation Time for Recurrent Neural Networks
- Training Large Language Models to Reason in a Continuous Latent Space
- Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation
- The Ignition Is Real, and It Lives at the Readout: Latent composition, difficulty-clocked ignition, and the interface-constituted commit in a recurrent-depth reasoner
- DeepLoop: Depth Scaling for Looped Transformers
- Retrofitting Recurrent Depth into a Pretrained Language Model: Installation, Extrapolation, Transfer, and Retention at Two Parameter Budgets
- Understanding Performance Collapse in Layer-Pruned Large Language Models via Decision Representation Transitions
- LLaMA: Open and Efficient Foundation Language Models
- When Fewer Layers Break More Chains: Layer Pruning Harms Test-Time Scaling in LLMs
- Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking
- When Does Recurrence Become an Algorithm? Convergence Selection in Weight-Tied Looped Transformers
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
- VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection