Spectrally Targeted Muon

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Spectrally Targeted Muon".

Jane: The gist The Muon optimizer orthogonalizes each update matrix, setting all of its singular values to one, and has proven highly effective for training large language models.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this new paper called Spectrally Targeted Muon from Vishrut Goyal and Rohan Ramkumar. Basically, they take the Muon optimizer—which is known for orthogonalizing every update matrix so all its singular values become one—and they try to figure out if that success comes from boosting tiny directions that gradient descent usually ignores or if it's actually suppressing big directions that mess up training.

Jane: That's the core question there. They introduce Spectrally Targeted Muon, which only orthogonalizes the singular values above or below a certain threshold tau, so you can slide between standard SGD and Muon by changing that tau. They use projections calculated by Newton-Schulz iteration on a shifted Gram matrix to do this, which is neat because it avoids needing Singular Value Decomposition entirely.

Lu: It’s interesting how they framed the mechanism there. They're essentially isolating the relevant subspaces using those projections based on the threshold tau so they don't mess with everything uniformly. They test this on CIFAR-ten and NanoGPT speedruns, tracking effective rank of gradients and updates, and a new metric called alignment of updates with the tangent space of the weight matrix’s isospectral manifold <ref:2610.10965#pg1,alignment of updates with the tangent space of the weight matrix’s>.

Meng: Tracking those metrics sounds like a way to see if the optimizer is actually learning something useful structurally rather than just making small mathematical adjustments that don't help the loss. What does this mean for practical application in training really big language models?

Lalam: I think it suggests that we can tailor the orthogonalization to target specific parts of the update matrix, instead of applying a blanket rule to every single dimension. It makes sense because large language models have huge parameter spaces where some directions are clearly more important than others.

Tom: Exactly. And they found three main things about this targeted approach. First, they said that the small singular values of the momentum aren't just noise; actually, orthogonalizing everything except those few largest singular values nearly matches Muon but only touches a tiny fraction of it, whereas if you only target the top ones, it falls short even though it captures almost all of what's there.

Jane: That’s a big distinction. It means for these language models where every momentum singular value is far below one, targeted orthogonalization can only amplify directions that are too small to train on at their raw scale, and Muon wins because it makes those tiny directions trainable in their raw form.

Lu: And the second thing they pointed out is that shrinking the largest singular values doesn't actually keep the parameter spectrum flat; that's just cheap, and it’s not what drives the loss curve down. It’s something else entirely that dictates where you end up.

Meng: So if shrinking those big singular values isn't the driver for better performance, then we need to focus on how we handle those smaller components instead of trying to flatten the whole spectrum with massive orthogonalization. That shifts our engineering focus significantly, doesn't it?

Lalam: Right. And they also found that AdamW differs from Muon mainly in how slowly it builds structure, which explains why AdamW starts slower and why warmup helps AdamW but hurts Muon. This suggests the difference is about the timeline of structural building up in the model parameters.

Tom: It’s clear then that Spectrally Targeted Muon isn't just another optimizer with a different setting; it’s a way to selectively apply orthogonalization based on which parts of the update matrix are most important for learning, and it gives us insights into what those important parts actually are.

Jane: So, in conclusion, they show that focusing the orthogonalization allows us to see if we're training on directions too small for standard methods or if we're accidentally suppressing dominant ones by applying full Muon across the board.

Lu: The implications for AI research is that understanding which singular values matter—the top few versus the bottom many—is crucial for designing effective update mechanisms, especially when dealing with models that have such high-dimensional parameter spaces.

Meng: From an engineering standpoint, if we can use these projections to guide our optimization steps without needing expensive SVDs every time, that would make training much more efficient in practice on large systems.

Lalam: And for the culture of how we build these models, it shows that precision in applying mathematical tools can lead to a deeper understanding of the underlying learning process than just blindly following a single algorithm.

Tom: Yeah, so this paper really gives us a sharper lens into how optimizers interact with the geometry of high-dimensional parameter spaces. It moves beyond just comparing final accuracy numbers and starts looking at the structure of the learning itself.

Conclusion: Tom: So we’re wrapping up on Spectrally Targeted Muon from Goyal and Ramkumar by talking about what this paper actually means for us right now.

Jane: Basically, they took the standard Muon optimizer—which makes every update matrix singular values equal one—and they figured out how to target only the important ones.

Lu: They introduce this Spectrally Targeted Muon method that lets you pick a threshold tau, so you can interpolate between standard SGD and the full Muon approach.

Meng: It uses projections calculated by Newton-Schulz iteration on a shifted Gram matrix instead of needing to calculate a full singular value decomposition every time, which is pretty efficient for large models.

Lalam: The core idea they found is that those small singular values in the momentum aren't just noise; they actually represent directions that the model can train on once they’re amplified.

Tom: That’s a big distinction because it means Muon wins when you have those tiny directions, and targeted methods are better when you want to be precise about which parts of the update matrix matter.

Jane: They found that shrinking the largest singular values doesn't actually keep the parameter spectrum flat; that’s just cheap and it doesn't drive the loss down on its own.

Lu: The third thing they noticed is how AdamW builds structure—it does so much slower than Muon, which explains why warmup helps AdamW but hurts Muon.

Meng: From an engineering standpoint, this targeted approach gives us a way to control exactly how much structure we’re forcing into the update step based on what the paper shows is actually relevant.

Lalam: I see this as a way for future AI to be more efficient by not treating every single direction in the weight matrix equally during updates.

Tom: Yeah, it shifts the focus from just getting high accuracy numbers to understanding where those high accuracy steps are actually coming from structurally.

Jane: And that leads us into how they actually proved this works on CIFAR-ten and NanoGPT speedruns, looking at things like effective rank of gradients and updates.

Lu: The implication is that understanding the spectrum—the distribution of those singular values—is a more important tool for designing optimizers than just tweaking the learning rate schedule.