DepthBench: Measuring How Residual Connections Enable More Computational Depth
cs.CV, cs.AI, cs.LG
Submitted: 2026-09-26
Updated: 2026-09-26
Code: https://github.com/allenai/OLMo-core
Terminology
Sources
- Program Synthesis with Large Language Models
- Layer Normalization
- Adaptive Input Representations for Neural Language Modeling
- GPT-NeoX-20B: An Open-Source Autoregressive Language Model
- Post-LayerNorm Is Back: Stable, ExpressivE, and Deep
- Nexus: Same Pretraining Loss, Better Downstream Generalization via Common Minima
- Evaluating Large Language Models Trained on Code
- Training Verifiers to Solve Math Word Problems
- Training Compute-Optimal Large Language Models
- Scaling Laws for Neural Language Models
- Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
- Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN
- SiameseNorm: Breaking the Barrier to Reconciling Pre/Post-Norm
- Inverse Depth Scaling From Most Layers Being Similar
- MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Decoupled Weight Decay Regularization
- LAuReL: Learned Augmented Residual Layer
- When Does Sparsity Mitigate the Curse of Depth in LLMs
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models