Stable initialization without the CLT
cs.LG
Submitted: 2026-09-24
Updated: 2026-09-24
Terminology
Sources
- JA-SIREN: Deterministic Initialization for Sinusoidal Networks via Spectral Matching
- Don't be lazy: CompleteP enables compute-efficient deep transformers
- COIN: COmpression with Implicit Neural representations
- How to Start Training: The Effect of Initialization and Architecture
- Deep Learning Scaling is Predictable, Empirically
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Deep Neural Networks as Gaussian Processes
- NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
- All you need is a good init
- Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
- Deep Information Propagation
- Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
- Learned Initializations for Optimizing Coordinate-Based Neural Representations
- Gaussian Pre-Activations in Neural Networks: Myth or Reality?
- Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks
- Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes
- Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
- A Spectral Condition for Feature Learning
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks