Understanding Transformer Optimization via Gradient Heterogeneity
Akiyoshi Tomihari, Issei Sato
cs.LG, cs.AI, cs.NE
Submitted: 2026-08-17
Updated: 2026-08-19
Terminology
Sources
- On Empirical Comparisons of Optimizers for Deep Learning
- The Geometry of Sign Gradient Descent
- What Does BERT Look At? An Analysis of BERT's Attention
- Longformer: The Long-Document Transformer
- Old Optimizer, New Norm: An Anthology
- Cherry on Top: Parameter Heterogeneity and Quantization in Large Language Models
- The Llama 3 Herd of Models
- DeepSeek-V3 Technical Report
- RoBERTa: A Robustly Optimized BERT Pretraining Approach
- Understanding Adam Requires Better Rotation Dependent Assumptions
- Mistral 7B
- Fine-Grained Visual Classification of Aircraft
- B2T Connection: Serving Stability and Performance in Deep Transformers
- Block-diagonal Hessian-free Optimization for Training Neural Networks
- On the representation of finite convex geometries with convex sets
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks