Attention Routing Stabilizes Early: Working-Set Inference for Recurrent Language Models
cs.CL, cs.LG
Submitted: 2026-09-23
Updated: 2026-09-27
Code: https://github.com/tbn5pj/WISE_code
Terminology
Sources
- PonderNet: Learning to Ponder
- PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling
- Training Verifiers to Solve Math Word Problems
- Universal Transformers
- Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference
- Depth-Adaptive Transformer
- Adaptive Computation Time for Recurrent Neural Networks
- Training Large Language Models to Reason in a Continuous Latent Space
- Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers
- Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence
- Mixture-of-Depths: Dynamically allocating compute in transformer-based language models
- Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
- Sharing Attention Weights for Fast Transformer
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering