When Dynamic Scaling Fails: A Scale-State Feedback Governs Recurrent-State Quantization in Linear-Attention and SSM LLMs

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "When Dynamic Scaling Fails".

Jane: The gist The mechanism is a feedback between the quantizer scale and the state it is derived from,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Let's talk about the title of "When Dynamic Scaling Fails: A Scale–State Feedback Governs Recurrent-State Quantization in Linear-Attention and SSM LLMs". It points directly at a feedback loop between the quantization scale and the state it's derived from.

Jane: So, instead of just saying dynamic scaling is bad, they're pointing to this specific mechanism as the cause of instability. It’s about how that scale changes based on what’s already in the state.

Lu: The authors are focusing on linear-attention and state-space models because they use a recurrent state, and quantizing that state at every step means errors compound inside the recurrence itself.

Meng: From an engineering standpoint, this suggests we can't just pick a bit-width once and assume it stays safe for a long decode; we have to account for how that scale interacts with the ongoing computation.

Lalam: The paper shows that when you apply INT4 per-token dynamic scaling, the closed-loop recurrent-state norm grows super-linearly with decode length, which is the runaway effect they are describing.

The paper's summary: Tom: So, what's the main finding here? They show that this feedback loop causes numerical runaway in per-token dynamic INT4 scaling, and they identified a way to cut that feedback edge by running FP and a per-token dynamic INT4 model in lockstep on the same tokens.

Jane: That lockstep intervention stops the numerical explosion, but they note that functional recovery is smaller than the numerical one; two thirds of the damage survives even when you stop the runaway.

Lu: The paper also says that causal identification via scale-yoking involves cutting this feedback edge directly by running FP and a per-token dynamic INT4 model in lockstep on the same tokens, which eliminates the numerical explosion.

Meng: That's interesting because it suggests an intervention exists to stabilize it, but we still have functional loss because of that intervention itself.

Lalam: The paper unifies this by showing that a persistent scale eliminates the feedback edge, or a calibrated ceiling bounds it, or you can do periodic resets to interrupt the loop.

The paper's improvements: Tom: They suggest four containment strategies: first, a persistent scale eliminates the feedback edge; second, a calibrated ceiling bounds it; third, periodic reset interrupts it; and fourth, using a higher bit-width just delays the loop.

Jane: Then they discuss allocation strategies too. They look at outlier-frequency channel targeting or exact sparse escape versus uniform group-wise quantization.

Lu: The paper finds that locating the outlier channels does not beat uniform group-wise quantization because the recurrent failure is a scale–state feedback to be contained rather than just a static set of outliers to be handled.

Meng: They point out that the best leak-free allocation is actually an extreme one—top three layers at INT8, and the rest at INT2, averaging four point zero bits—which they say is a genuine Pareto improvement over uniform group-wise quantization at a matched budget <ref:2610.15139#pg2>.

Lalam: Uniform group-wise scaling beats all three containment interventions because it’s the least-damaging uniform policy on these models, according to their audit §eight <ref:2610.15139#pg3>.

Conclusion: Tom: To wrap up, the paper says that a persistent scale eliminates the feedback edge, a calibrated ceiling bounds it, periodic reset interrupts it, and a higher bit-width merely delays the loop. They prefer any persistently calibrated scale over per-token dynamic scaling.

Jane: The key reporting point they want us to carry forward is this: "a bit-width certified at five hundred twelve steps says little about 4K and nothing about 32K, and the horizon is where low-bit recurrent-state inference actually breaks."

Lu: They also stress that the best policy short is the best policy long because short evaluation badly underestimates risk, as the good policy has no short-window events to fit.

Meng: From a deployment view, this means we shouldn't trust a bit-width certification at a short decode length if we plan on running long contexts.

Lalam: The paper is an empirical analysis of numerical stability in quantized inference, warning against certifying a quantizer at short decode lengths because it can overstate deployed reliability.

Tom: So, the title "When Dynamic Scaling Fails: A Scale–State Feedback Governs Recurrent-State Quantization in Linear-Attention and SSM LLMs" is really about understanding that internal feedback loop in recurrent state quantization.

Jane: It's a warning to move away from just looking at a single bit-width number and start looking at the entire decode horizon for stability.