Give it Space! Explicit Disentangling of Positional and Semantic Representations in Encoders
cs.CL, cs.AI
Submitted: 2026-05-28
Updated: 2026-09-02
Code: https://github.com/LequeuISIR/DSTG-encoder
Terminology
Sources
- ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
- Contextual Position Encoding: Learning to Count What's Important
- Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation
- Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
- Rethinking Positional Encoding in Language Pre-training
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
- Functional Interpolation for Relative Positions Improves Long Context Transformers
- GLU Variants Improve Transformer
- Uncovering hidden geometry in Transformers via disentangling position and context
- RoBERTa: A Robustly Optimized BERT Pretraining Approach
- RoFormer: Enhanced Transformer with Rotary Position Embedding
- Decoupled Weight Decay Regularization
- Decoupling Positional and Symbolic Attention Behavior in Transformers
- Pointer Sentinel Mixture Models
- YaRN: Efficient Context Window Extension of Large Language Models
- Retrieval Head Mechanistically Explains Long-Context Factuality
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering