No More Free Lunch: Corpus Task Complexity Matters as Corpora Grow

arXiv:2609.29245 · cs.CL, cs.AI · Submitted 2026-09-24 · Read on arXiv

cs.CL, cs.AI

Submitted: 2026-09-24

Updated: 2026-09-24

Terminology

• Sparse Attention — 26× in this paper · explained in 5 episodes
• Benchmark — 15× in this paper · explained in 2 episodes
• Generalization — 10× in this paper · explained in 13 episodes
• Corpus Task Complexity — 8× in this paper
• Aggregation — 6× in this paper · explained in 1 episode
• Transformer — 5× in this paper · explained in 2 episodes
• Fine-Tuning — 4× in this paper · explained in 5 episodes
• Hybrid Architectures — 4× in this paper · explained in 3 episodes
• Actor — 2× in this paper · explained in 3 episodes
• Core — 2× in this paper · explained in 2 episodes
• In-Context Learning — 2× in this paper · explained in 4 episodes
• Redundancy — 2× in this paper · explained in 2 episodes
• Trace — 2× in this paper · explained in 2 episodes
• Activation Checkpointing — 1× in this paper · explained in 1 episode
• Agentic System — 1× in this paper · explained in 2 episodes
• Attention Mechanism — 1× in this paper · explained in 3 episodes
• Few-Shot Learning — 1× in this paper · explained in 4 episodes
• Fidelity — 1× in this paper · explained in 2 episodes
• Generative Model — 1× in this paper · explained in 2 episodes
• Generative Models — 1× in this paper · explained in 2 episodes
• KV cache — 1× in this paper · explained in 3 episodes
• Positional Bias — 1× in this paper · explained in 2 episodes

Sources

• Star Attention: Efficient LLM Inference over Long Sequences
• LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks
• Longformer: The Long-Document Transformer
• Oolong: Evaluating Long Context Reasoning and Aggregation Capabilities
• LongBench Pro: A More Realistic and Comprehensive Bilingual Long-Context Evaluation Benchmark
• Overview of the TREC 2021 deep learning track
• AbsenceBench: Language Models Can't Tell What's Missing
• Is It Really Long Context if All You Need Is Retrieval? Towards Genuinely Difficult Long Context NLP
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoders
• Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
• PubMedQA: A Dataset for Biomedical Research Question Answering
• Dense Passage Retrieval for Open-Domain Question Answering
• Randomized YaRN Improves Length Generalization for Long-Context Reasoning
• Olmo Hybrid: From Theory to Practice and Back
• YaRN: Efficient Context Window Extension of Large Language Models
• OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and concepts
• DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processing
• BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval
• OBLIQ-Bench: Exposing Overlooked Bottlenecks in Modern Retrievers with Latent and Implicit Queries

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering