Sparse Competition during Training For the Emergence of Specialized Modules
cs.LG
Submitted: 2026-08-31
Updated: 2026-08-31
Code: https://github.com/BabaVegato/Minimal-SCM
Terminology
Sources
- Gradient Routing: Masking Gradients to Localize Computation in Neural Networks
- Sparse Autoencoders Find Highly Interpretable Features in Language Models
- Toy Models of Superposition
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
- Studying Cross-cluster Modularity in Neural Networks
- Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks