Shifting Mechanisms: How Positional Encoding Choice Shapes In-Context Retrieval
cs.CL, cs.LG
Submitted: 2026-09-29
Updated: 2026-09-29
Code: https://github.com/ericenouen/shiftmech
Terminology
Sources
- Round and Round We Go! What makes Rotary Positional Encodings useful?
- Longformer: The Long-Document Transformer
- NVIDIA Nemotron 3: Efficient and Open Intelligence
- Context Length Alone Hurts LLM Performance Despite Perfect Retrieval
- RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably
- Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings
- Gemma 4 Technical Report
- Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddings
- The Llama 3 Herd of Models
- RULER: What's the Real Context Size of Your Long-Context Language Models?
- Gemma 3 Technical Report
- Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPE
- Kimi Linear: An Expressive, Efficient Attention Architecture
- Rotary Positional Embeddings as Phase Modulation: Theoretical Bounds on the RoPE Base for Long-Context Transformers
- Interpretability Can Be Actionable
- Language Models use Lookbacks to Track Beliefs
- SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modeling
- Rethinking the Role of Efficient Attention in Hybrid Architectures
- Qwen3.5-Omni Technical Report
- Gemma 2: Improving Open Language Models at a Practical Size
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering