Stochastic Parroting in Temporal Attention -- Regulating the Diagonal Sink

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Stochastic Parroting in Temporal Attention -- Regulating the Diagonal Sink".

Jane: The gist The authors provide a theoretical characterization of the Jacobian of Temporal Attention to show that temporal attention layers suffer from a diagonal attention sink,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we’re looking at why these temporal attention layers suffer from this diagonal attention sink and what the authors propose to fix it in this paper. It’s about managing how information moves across time within a sequence.

Jane: The core idea here is that they theoretically characterize the Jacobian of temporal attention to show that it leads to stochastic parroting, which means the model starts repeating itself or losing distinct temporal patterns in a way that gets worse with more data points.

Lu: They derive the uniform expectation of the Jacobian norm over sequence length j and find that for off-diagonal terms, those non-local signal strengths decay as one/T, where T is the sequence length <ref:2602.10956#pg2>.

Meng: So they’re saying that the information between different time steps, the off-diagonal stuff, fades out over time. That sounds like a problem for capturing long-term dependencies if you don't handle it right.

Lalam: And the paper points out that while those off-diagonal terms decay nicely, the diagonal terms have a larger bound because they also include this query term's bound CQ/sqrt d k <ref:2602.10956#pg3>.

Tom: So there's a mismatch: the non-local information shrinks, but the direct self-attention on the same token gets a bigger bound because of that query part.

Jane: And to address this imbalance, they suggest three specific regularization methods to reduce that diagonal attention sink. These are a diagonal mask with entries of negative infinity and others zero, standard dropout applied only to the diagonal of the attention matrix, and adding a negative scalar penalty to the raw attention scores.

The paper's summary: Tom: The authors summarize by showing that these three regularization methods actually work in practice. They test them against models without residual connections and without any regularization at all, and they find that the diagonal mask gives results similar to the version without regularization <ref:2602.10956#pg3>.

Jane: It’s interesting because they found that standard dropout or adding a penalty actually show a noticeable improvement, around two point five percent better than the non-regularized versions in their experiments Table one.

Lu: The paper suggests that the diagonal mask suppresses the query path of the softmax attention, which results in attention emphasizing specific keys for the whole sequence, while dropout and penalty allow for a more nuanced control over that diagonal behavior <ref:2602.10956#pg3>.

Meng: So they’re suggesting that you don't have to do a full mask; applying just the diagonal entries being negative infinity is enough to mitigate this parroting effect, and dropout or penalty gives you more control over the specific keys attending to specific queries Table one <ref:2602.10956#pg1>.

Lalam: And they specifically found that for long horizons, like predicting thirty-six future steps, the diagonal penalty actually gave the best results in their evaluation <ref:2602.10956#pg3>.

The paper's improvements: Tom: So what are the actual suggested improvements from this paper? It’s not just about finding a problem; it’s about fixing it with architectural changes.

Jane: They suggest using a diagonal mask with negative infinity for the diagonal entries and zero for everything else, which they think is effective at mitigating that diagonal attention sink <ref:2602.10956#pg3>.

Lu: They also bring up the idea of residual connections, suggesting they should be used because they remain expressible even when the diagonal is completely removed, implying that the diagonal part provides relatively less important information <ref:2602.10956#pg3>.

Meng: If you're deploying this at scale, I’m interested in the residual connection idea because it means self-information stays stable regardless of sequence length, which is good for model consistency Table one <ref:2602.10956#pg1>.

Lalam: And they also tested a negative scalar penalty added to raw attention scores. They found that this method is superior to the mask or dropout for long-horizon forecasting, allowing the model to use those diagonal entries when it’s beneficial <ref:2602.10956#pg3>.

Conclusion: Tom: So wrapping up, they prove the existence of these diagonal attention sinks induced by the residual and the query weight matrix in temporal attention layers, and that this leads to stochastic parroting that intensifies with sequence length.

Jane: The main point is that while full diagonal masks suppress the query path of softmax attention, leading to strong emphasis on specific keys, using a diagonal penalty or dropout significantly improves performance over just using a full mask <ref:2602.10956#pg3>.

Lu: Overall, as sequence length T gets larger, the off-diagonal entries naturally shrink to O(one/T), but the diagonal term keeps its influence because of that query bound <ref:2602.10956#pg3>.

Meng: So for production, if you want the best results on long horizons, they recommend using that negative scalar penalty method because it allows you to capture more varied patterns Table one <ref:2602.10956#pg1>.

Lalam: I'm confident in this direction; the analysis shows how off-diagonal attention scores depend on sequence length, which gives us a clear path for making informed architectural adjustments to prevent over-squashing when mixing temporal and spatial processing <ref:2602.10956#pg1>.

Tom: So we’ve looked at the Stochastic Parroting in Temporal Attention -- Regulating the Diagonal Sink paper, showing that by regulating this sink, especially with penalties for long horizons, we can improve temporal attention performance even as sequences grow larger.