Beyond Semantics: How Temporal Biases Shape Retrieval in Transformer and State-Space Models

In-context learning in Large Language Models (LLMs) is governed by both temporal and semantic relationships, shaping how they retrieve contextual information.

The gist: Models consistently place the highest probabilities on tokens following a repeated token, but with a notable bias for those nearest the beginning or end of the input.

Experiment 1: Isolating Temporal Positional Biases

This experiment investigated inherent temporal biases in LLM retrieval, independent of semantic content, by prompting models with sequences containing multiple presentations of the same tokens. The methodology involved fixing the positions of repeated tokens and permuting all others to isolate temporal effects on next-token prediction. Results showed that across diverse models and architectures (transformers and state-space models), there was a consistent preference for retrieving information linked to tokens presented near the beginning or end of the context, mirroring primacy and recency effects observed in human memory studies.

Key Findings from Experiment 1:

** Both Transformer and SSM models show a tendency for serial recall (predicting the '+1' tokens). This preference is modulated by position, showing strong primacy/recency effects. For example, Mistral shows a recency bias, while Falcon-Mamba shows a primacy bias. Gemma’s preference shifts: with fewer repetitions, the peak is mid-prompt, but with more repetitions, it shifts towards the end.**

** The strength of this recall is modulated by position within the prompt. For instance, in Figure 3 (Experiment 1), probabilities for the ‘+1’ token vary depending on its repetition number (depth in prompt).**

Experiment 2: Testing Episodic Retrieval with Interference

This experiment assessed models’ ability to retrieve specific temporal sequences ('episodes') when presented alongside other partially overlapping sequences, testing their capacity for temporal separation. Prompts contained five distinct episodes, each defined by a unique ‘context token’, followed by the ‘fixed token’ and a unique ‘target token’. The model was evaluated on predicting the next token following a probe pair (e.g., 'XA') to distinguish the correct episode from others sharing the fixed token.

Key Findings from Experiment 2:

** Most models successfully retrieved the correct episode, but retrieval was strongest for episodes near the end of the prompt (recency bias). Mamba and Falcon-Mamba models showed less robust retrieval than others.**

** Beyond the highest peak for the target episode, smaller peaks corresponding to non-probed episodes were often visible, indicating interference or similarity matching based on the shared fixed token ‘A’. The magnitude of the correct target peak often varies with position, generally being strongest for episodes nearer the end of the prompt.**

Mechanistic Insights via Ablation Studies

To understand these temporal effects, an ablation study focused on induction heads in transformer models. Induction heads were identified as crucial components that operate by finding previous occurrences of a current token and attending to the token that followed it, effectively learning and reproducing sequences based on temporal association.

Key Findings from Ablation Studies:

** Ablating top induction heads consistently and significantly degrades the ‘+1’ token probability peaks, particularly with 50 or 100 heads ablated. This confirms the critical role of induction heads in the serial recall behavior.**

** In Experiment 2, ablating induction heads often disrupted the model’s ability to selectively retrieve the single target episode, leading to probabilities becoming more distributed across potential target tokens from different episodes, indicating increased interference.**

Architectural Comparison and Implications

The study found that while transformer models showed a mechanistic link between temporal biases and induction heads, state-space models (SSMs) exhibited comparable temporal biases. SSMs demonstrated a similar ‘+1’ token preference, suggesting that basic sequence copying or pattern completion might be a convergent capability learned by different sequence modeling architectures. The paper suggests these temporal biases may arise from fundamental properties of sequential data processing, such as the influence of positional encodings or inherent limitations in how architectures maintain and access information over long distances. These findings imply that addressing the “lost in the middle” problem requires tackling fundamental temporal processing limitations, potentially related to positional information or state management, which may persist even in non-attention architectures. Furthermore, results suggest that for tasks requiring retrieval based on relative temporal position and handling interference, SSMs exhibit functionally similar limitations to transformers.

Limitations

The study primarily used token-random prompts rather than natural text and analyzed next-token probabilities. Future work is suggested to build upon this understanding of pure temporal biases to explore the more complex interplay between ‘when’ something was said and ‘what’ was said in rich, semantic contexts. The authors also noted that no globally distinct temporal patterns consistently separated transformer and state-space architectures.

References

(A full list of references is provided at the end of the paper.)

**(Self-Correction/Review: The summary adheres to the structural requirements:

Here are the specific improvements for AI systems based on this research, categorized by the capabilities they enable:

) Improved System Capabilities: Temporal Context Separation and Episodic Memory Simulation

The core improvement is enabling LLMs to perform robust episodic retrieval, meaning they can distinguish between temporally separated events even when they share semantic content. This moves the model beyond simple pattern matching to a level of contextual memory management analogous to human episodic memory.

Specific improvements include:

2. Improved Episodic Retrieval in Contextual Reasoning: The system will be significantly better at answering complex, multi-step queries that require recalling specific details from different points in a long input context (e.g., Given the initial plan, what was the specific constraint mentioned three steps later?).

4. Temporal Bias Mitigation and Robustness: The system will be less susceptible to positional biases (lost in the middle effect) when retrieving information from long contexts. It will maintain high recall reliability for information presented at both the very beginning and very end of a prompt, regardless of where it is located in the middle.

6. Enhanced Serial Recall and Pattern Completion: The system will exhibit stronger serial recall, meaning if a sequence is presented multiple times, it will be more likely to correctly predict the token immediately following the previous instance (the "+1" token), leading to better performance on sequential generation tasks and code completion where context repetition is common.

8. Architectural Agnosticism: The system's improved temporal biases will not be strictly dependent on a specific architecture (Transformer vs. State-Space Model). This suggests that training or fine-tuning methodologies can be designed to induce these fundamental temporal processing capabilities in diverse sequence models, making the AI more versatile across different hardware and model families.

10. Mechanistic Understanding and Debugging: By explicitly linking performance to induction heads (in transformers) or state compression dynamics (in SSMs), developers gain a diagnostic tool. If an AI fails at episodic retrieval, researchers can pinpoint whether the failure is due to insufficient attention head capacity or limitations in the model's state management mechanism.