Randomized YaRN Improves Length Generalization for Long-Context Reasoning

arXiv:2606.23687 · cs.CL · Submitted 2026-06-22 · Read on arXiv

cs.CL

Submitted: 2026-06-22

Updated: 2026-08-30

Code: https://github.com/Manas-Mehta/Randomized-YaRN

Terminology

• Generalization — 32× in this paper · explained in 13 episodes
• Fine-Tuning — 11× in this paper · explained in 5 episodes
• Benchmark — 9× in this paper · explained in 2 episodes
• Transformer — 6× in this paper · explained in 2 episodes
• Accuracy — 4× in this paper · explained in 1 episode
• Ablation — 3× in this paper · explained in 1 episode
• Large Language Models (LLMs) — 3× in this paper · explained in 6 episodes
• Multi-hop Reasoning — 3× in this paper · explained in 3 episodes
• Randomized Positional Encoding — 3× in this paper
• International Conference on Learning Representations — 2× in this paper
• LoRA Fine-Tuning — 2× in this paper · explained in 3 episodes
• Ablation Studies — 1× in this paper · explained in 1 episode
• Absolute Indices — 1× in this paper · explained in 1 episode
• Adaptation — 1× in this paper · explained in 1 episode
• Alignment — 1× in this paper · explained in 1 episode
• Attention Mechanism — 1× in this paper · explained in 2 episodes
• BABILong and Multi-Round Coreference Resolution — 1× in this paper
• Benchmarking — 1× in this paper · explained in 3 episodes
• Embeddings — 1× in this paper · explained in 2 episodes
• Foundation Model — 1× in this paper · explained in 3 episodes
• Foundation Models — 1× in this paper · explained in 7 episodes
• Transformer Architecture — 1× in this paper · explained in 2 episodes
• Transformer Models — 1× in this paper · explained in 2 episodes

Sources

• Extending Context Window of Large Language Models via Positional Interpolation
• The Llama 3 Herd of Models
• BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack
• Olmo 3
• LongSkywork: A Training Recipe for Efficiently Extending Context Length in Large Language Models
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Qwen2.5 Technical Report
• Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries
• Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering