Dead Directions: Geometric Singular Learning
cs.LG, stat.ML
Submitted: 2026-06-04
Updated: 2026-09-09
Comments: v2: substantially revised. 176 pages, 13 figures, 14 tables. New machine-checked appendix (Lean 4 sources as ancillary files); singular-fluctuation theorem restated and its strict inequality proved; determinantal RLCT closed at depth <= 3; multi-component recovery to r = 5. Corrections to the attention-chain rate, SwiGLU composition, the LayerNorm bracket, and the curvature-rate chain
License: http://creativecommons.org/licenses/by/4.0/
The gist: Singular learning theory and information geometry study the same spaces: the former in resolved coordinates, the latter in original coordinates under a non-degeneracy assumption that
Terminology
Abstract
Singular learning theory and information geometry study the same spaces: the former in resolved coordinates, the latter in original coordinates under a non-degeneracy assumption that overparameterised models violate. This paper carries one direction of the bridge between them, from Watanabe's invariants to Fisher geometry, through one primitive, the dead direction: a unit vector along which the Fisher metric degenerates, equivalently a direction crossing the analytic singular set along which the KL divergence keeps a zero of high order, its KL order set by how fast that divergence vanishes. Our central result recovers the KL order as the decay rate of the directional Fisher quadratic form approaching the singularity, in original coordinates, without a Hironaka resolution. A selection rule on smooth fibres translates this rate into Watanabe's single-direction contribution to the real log canonical threshold, and the recovery extends to multi-component crossings, multiplicity m, the singular fluctuation ν, prior-RLCT shifts, and tempered posteriors. We then carry the rate into a deep network: a multi-layer K-FAC factorisation writes each Fisher block as a product of activation- and gradient-side rates with a duality between them, instantiated at residual streams, layer normalisation, and attention. A quotient theorem carries the rate to the gauge quotient for optimizers whose update commutes with the group action; Adam's per-coordinate preconditioner fails that condition, so we construct DDCAdam, an equivariant Adam-family preconditioner, and prove the quotient rate along its trajectory. The result is a trajectory-rate readout of Watanabe's triple (λ, m, ν) from one checkpoint's forward and backward passes, without posterior sampling.
Sources
- The Loss Kernel: A Geometric Probe for Deep Learning Interpretability
- Structural Inference: Interpreting Small Language Models with Susceptibilities
- Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit
- Modes of Sequence Models and Learning Coefficients
- Dynamical versus Bayesian Phase Transitions in a Toy Model of Superposition
- The Z-loss: a shift and scale invariant classification loss belonging to the Spherical Family
- How do simple rotations affect the implicit bias of Adam?
- Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth
- Functional Equivalence and Path Connectivity of Reducible Hyperbolic Tangent Networks
- Towards Spectroscopy: Susceptibility Clusters in Language Models
- Loss Landscape Degeneracy and Stagewise Development in Transformers
- Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
- Bayesian Influence Functions for Hessian-Free Data Attribution
- Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics
- Limitations of the Empirical Fisher Approximation for Natural Gradient Descent
- Influence Dynamics and Stagewise Data Attribution
- Optimizing Neural Networks with Kronecker-factored Approximate Curvature
- Programs as Singularities
- Progress measures for grokking via mechanistic interpretability
- Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks