Attention Amnesia in Hybrid LLMs: When CoT Fine-Tuning Breaks Long-Range Recall, and How to Fix It

arXiv:2606.11052 · cs.CL · Submitted 2026-06-09 · Read on arXiv

cs.CL

Submitted: 2026-06-09

Updated: 2026-08-30

Code: https://github.com/LARK-AI-Lab/QK-Restore

Terminology

• Fine-Tuning — 36× in this paper · explained in 5 episodes
• Hybrid Models — 16× in this paper · explained in 2 episodes
• Transformer — 9× in this paper · explained in 2 episodes
• Distillation — 7× in this paper · explained in 2 episodes
• Spectral Gap — 6× in this paper · explained in 5 episodes
• Accuracy — 5× in this paper · explained in 1 episode
• Adaptation — 5× in this paper · explained in 1 episode
• Reasoning Traces — 5× in this paper · explained in 3 episodes
• Trace — 5× in this paper · explained in 2 episodes
• Ablation — 3× in this paper · explained in 1 episode
• Benchmark — 3× in this paper · explained in 2 episodes
• Decoupling — 3× in this paper · explained in 2 episodes
• Chain-of-Thought (CoT) — 2× in this paper · explained in 4 episodes
• Hybrid Architectures — 2× in this paper · explained in 3 episodes
• Redundancy — 2× in this paper · explained in 2 episodes
• Supervised Fine-Tuning (SFT) — 2× in this paper · explained in 8 episodes
• Ablation study — 1× in this paper · explained in 1 episode
• Abundance — 1× in this paper · explained in 1 episode
• Catastrophic Forgetting — 1× in this paper · explained in 5 episodes
• Instruction Tuning — 1× in this paper · explained in 2 episodes
• Latent Reasoning — 1× in this paper · explained in 2 episodes
• Reinforcement Learning — 1× in this paper · explained in 4 episodes
• Robustness — 1× in this paper · explained in 5 episodes
• Speculative Decoding — 1× in this paper · explained in 2 episodes
• Structured Reasoning — 1× in this paper · explained in 3 episodes
• Verifier — 1× in this paper · explained in 2 episodes

Sources

• SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model
• Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes
• Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contexts
• Training Verifiers to Solve Math Word Problems
• Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
• RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scale
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• OpenThoughts: Data Recipes for Reasoning Models
• RAD: Redundancy-Aware Distillation for Hybrid Models via Self-Speculative Decoding
• RULER: What's the Real Context Size of Your Long-Context Language Models?
• Repeat After Me: Transformers are Better than State Space Models at Copying
• Mistral 7B
• Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
• MiroMind-M1: An Open-Source Advancement in Mathematical Reasoning via Context-Aware Multi-Stage Policy Optimization
• Distilling to Hybrid Attention Models via KL-Guided Layer Selection
• TL;DR: Too Long, Do Re-weighting for Efficient LLM Reasoning Compression
• Let's Verify Step by Step
• NVIDIA Nemotron 3: Efficient and Open Intelligence
• Training language models to follow instructions with human feedback
• Why think step by step? Reasoning emerges from the locality of experience

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering