Where You Place the Norm Matters: From Prejudiced to Neutral Initializations
cs.LG, cond-mat.dis-nn
Submitted: 2025-05-16
Updated: 2026-04-02
Code: https://github.com/EmanueleFrancazi/IGB-and-Normalization
Terminology
Sources
- Layer Normalization
- Batch Normalization Explained
- When Bias Meets Trainability: Connecting Theories of Initialization
- CNN Is All You Need
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Emergence of Globally Attracting Fixed Points in Deep Neural Networks With Nonlinear Activations
- All you need is a good init
- On the Nonlinearity of Layer Normalization
- Deep Information Propagation
- Density of states in neural networks: an in-depth exploration of learning in parameter space
- A Mean Field Theory of Batch Normalization
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks