Model Capacity Determines Grokking through Competing Memorisation and Generalisation Speeds
summary
The gist
The gist The onset of grokking coincides closely with the parameter count at which two measurable learning timescales, memorisation speed and generalisation speed, intersect, which is strictly larger
In short
The paper investigates how model capacity dictates 'grokking'—the ability to solve problems by memorizing data rather than learning general rules—on modular arithmetic tasks. It finds that grokking occurs when a model's parameter count reaches the intersection point between two competing learning speeds: memorization speed and generalization speed. This mechanism explains grokking as a competition between fitting data quickly and finding robust solutions.
Key concepts
- Memorisation Speed (Tmem(P))
- This measures how many training epochs it takes for a model to perfectly memorize a random dataset of equivalent information content. It quantifies the speed at which the model can simply store the input data, regardless of whether it learns underlying rules or not.
- Generalisation Speed (Tgen(P))
- This measures the number of epochs required for a model to achieve high accuracy on a modular arithmetic task using validation data. It represents how quickly the model develops robust, generalizable solutions rather than just memorizing specific examples.
- Grokking Regime
- This is the state where grokking occurs. It happens when the parameter count is beyond the point where memorization first becomes possible, and it specifically occurs when memorization speed slows down relative to generalization speed, meaning the model starts memorizing before it can reliably generalize.
- Information Capacity (Cap d(Θ))
- This formalizes how much information a model's parameters can hold. It is approximated by fitting a linear model to empirical capacity constants, linking the number of parameters directly to the total bits of information the model can potentially represent.
Terminology used across episodes
This episode discusses
- Model Capacity Determines Grokking through Competing Memorisation and Generalisation Speeds · Paper Radio
- Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws
- Information-Theoretic Progress Measures reveal Grokking is an Emergent Phase Transition
- Unifying Grokking and Double Descent
- The Complexity Dynamics of Grokking
- Towards Empirical Interpretation of Internal Circuits and Properties in Grokked Transformers on Modular Polynomials
- Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks
- Unified View of Grokking, Double Descent and Emergent Abilities: A Perspective from Circuits Competition
- Grokking as the Transition from Lazy to Rich Training Dynamics
- Grokfast: Accelerated Grokking by Amplifying Slow Gradients
- Omnigrok: Grokking Beyond Algorithmic Data
- Decoupled Weight Decay Regularization
- Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking
- A Systematic Empirical Study of Grokking: Depth, Architecture, Activation, and Regularization
- A Tale of Two Circuits: Grokking as Competition of Sparse and Dense Subnetworks
- How much do language models memorize?
- Deep Double Descent: Where Bigger Models and More Data Hurt
- Progress measures for grokking via mechanistic interpretability
- Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets
- How Much Knowledge Can You Pack Into the Parameters of a Language Model?
- Grokking as a First Order Phase Transition in Two Layer Networks
The paper
Model Capacity Determines Grokking through Competing Memorisation and Generalisation Speeds · Read on arXiv
Yiding Song, Hanming Ye
Harvard College
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: Today's paper: "Model Capacity Determines Grokking through Competing Memorisation and Generalisation Speeds".
Jane: The gist The onset of grokking coincides closely with the parameter count at which two measurable learning timescales, memorisation speed and generalisation speed, intersect,
Tom: First, who's behind it and why it matters.
Paper summary: Jane: So, to recap what we just heard, this paper, "Model Capacity Determines Grokking through Competing Memorisation and Generalisation Speeds," argues that grokking isn't a simple scaling problem with model size. It introduces an information-theoretic framework where grokking is the result of a competition between memorisation speed, Tmem(P), and generalisation speed, Tgen(P) <ref:2605.09724#pg1>.
Tom: They estimate these speeds on random-label data of equivalent complexity for Tmem, and on the modular task itself for Tgen. The core claim is that grokking emerges close to the parameter scale where these two timescales cross <ref:2605.09724#pg1>.
Lu: They define memorisation speed as the number of epochs required to fit a random dataset with the same total number of bits as the modular arithmetic dataset, and generalisation speed is how long it takes for validation accuracy to exceed a saturation threshold <ref:2605.09724#pg1>.
Meng: It’s like comparing two runners in a race. One runner memorises everything quickly, the other learns the actual rules of the game; grokking happens when those two times are very close <ref:2605.09724#pg1>.
Jane: The paper tests this by sweeping parameter count P and measuring these speeds, showing that in a grokking regime beyond the speed intersection, memorisation actually happens first, followed by generalisation <ref:2605.09724#pg1>.
Tom: They also showed strong predictive power in their sweep tests, with a Spearman’s rho of zero point nine seven six and Kendall’s tau of zero point nine one one <ref:2605.09724#pg3>.
Lu: This means the intersection point they predict is a very reliable marker for when grokking will happen in practice on this type of task <ref:2605.09724#pg3>.
Meng: What does this mean practically? It suggests that instead of just focusing on model size alone, we should be looking at the dynamic balance between how quickly a model can store information versus how quickly it can actually solve the underlying mathematical structure <ref:2605.09724#pg1>.
Jane: It shifts the focus from static capacity to dynamic learning processes that determine this transition <ref:2605.09724#pg1>.
Conclusion: Tom: So, wrapping up this discussion on "Model Capacity Determines Grokking through Competing Memorisation and Generalisation Speeds," the authors are making a strong point about the mechanism behind grokking. They’re not just observing that large models grok and saying it’s because they are big <ref:2605.09724#pg1>.
Jane: They propose that the crucial factor is this competition between Tmem(P) and Tgen(P). The intersection of these two speeds predicts the onset of grokking within a fixed hyperparameter setting, which is something they tested across different training setups <ref:2605.09724#pg1>.
Lu: They emphasize that this prediction doesn't require a single calibration constant to work everywhere; it just shifts smoothly depending on things like weight decay or the depth of the model <ref:2605.09724#pg1>.
Meng: From an engineering standpoint, this is useful because it tells us what we’re actually tuning matters more than just chasing a single size number <ref:2605.09724#pg1>.
Tom: Essentially, the paper gives us a mechanistic account of how model capacity controls grokking by setting the relative speeds of those two learning processes <ref:2605.09724#pg1>.
Jane: The implication for understanding large models is that we need to look at these competing timescales to predict when and why delayed generalisation will occur <ref:2605.09724#pg1>.
Lu: This framework opens up a new way to analyze overparameterised optimisation by looking at the interplay between fitting data and finding solutions <ref:2605.09724#pg3>.
Meng: It’s less about brute force size and more about balancing these two opposing forces during training <ref:2605.09724#pg1>.
More episodes
- 2510.00375-Multidimensional Bayesian Active Machine Learning of Working Memory Task Performance
- 2510.03151-Mixture of Many Zero-Compute Experts: A High-Rate Quantization Theory Perspective
- 2510.15714-Second-Order Optimization with Asynchronous Curvature
- 2512.02444-MetaJoin: Adaptive Join Selections for Heterogeneous Tables
- 2602.01149-A Generalizable Machine Learning Framework for Half-Heusler Thermoelectrics
- 2602.06824-RanSOM: Second-Order Momentum with Randomized Scaling for Constrained and Unconstrained Optimization
- Daily Summary for 2026-10-15
- 2601.22318-Federate the Router: Learning Language Model Routers with Sparse and Decentralized Evaluations
- 2602.08617-ERIS: Enhancing the Privacy-Utility-Scalability Trade-Off in Federated Learning via Federated Shard Aggregation
- 2603.00530-Bridge Matching Sampler: Scalable Sampling via Generalized Fixed-Point Diffusion Matching