How Local Mixing Encodes Relative Position in Global NoPE Attention
cs.CL, cs.AI, cs.LG
Submitted: 2026-09-29
Updated: 2026-09-29
Terminology
Sources
- Round and Round We Go! What makes Rotary Positional Encodings useful?
- How to Train Long-Context Language Models (Effectively)
- Gemma 2: Improving Open Language Models at a Practical Size
- Kimi K3: Open Frontier Intelligence
- Functional Interpolation for Relative Positions Improves Long Context Transformers
- Forgetting Transformer: Softmax Attention with a Forget Gate
- Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
- SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modeling
- RoFormer: Enhanced Transformer with Rotary Position Embedding
- Scaling Stick-Breaking Attention: An Efficient Implementation and In-depth Study
- On the Emergence of Position Bias in Transformers
- Group Representational Position Encoding
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering