HySPE: Positional Encoding via Symplectic Dual Shears

arXiv:2610.10154 · cs.CL · Submitted 2026-10-07 · Read on arXiv

cs.CL

Submitted: 2026-10-07

Updated: 2026-10-07

Code: https://github.com/karpathy/char-rnn

Terminology

• Perplexity — 17× in this paper · explained in 3 episodes
• Transformer — 13× in this paper · explained in 2 episodes
• Benchmark — 3× in this paper · explained in 2 episodes
• Embeddings — 3× in this paper · explained in 2 episodes
• Attention Entropy — 2× in this paper · explained in 2 episodes
• Benchmarking — 2× in this paper · explained in 4 episodes
• Fidelity — 2× in this paper · explained in 2 episodes
• Attention Mechanism — 1× in this paper · explained in 3 episodes
• Entropy Collapse — 1× in this paper · explained in 2 episodes
• Foundation Model — 1× in this paper · explained in 3 episodes
• Foundation Models — 1× in this paper · explained in 7 episodes
• Generalization — 1× in this paper · explained in 13 episodes
• Grounding — 1× in this paper · explained in 3 episodes
• Representation Learning — 1× in this paper · explained in 2 episodes
• Robustness — 1× in this paper · explained in 5 episodes
• Trace — 1× in this paper · explained in 2 episodes
• Transformer Architecture — 1× in this paper · explained in 2 episodes

Sources

• RoFormer: Enhanced Transformer with Rotary Position Embedding
• The Impact of Positional Encoding on Length Generalization in Transformers
• Extending Context Window of Large Language Models via Positional Interpolation
• Ananke: Contractive Torus Attractor Networks
• Minkowski Attractor Networks: Closed-Form Hyperbolic Flows for Visual Representations
• HoPE: Hyperbolic Rotary Positional Encoding for Stable Long-Range Dependency Modeling in Large Language Models
• Retentive Network: A Successor to Transformer for Large Language Models
• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• Pointer Sentinel Mixture Models

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering