Self-Evolving Search Index
cs.IR, cs.AI
Submitted: 2026-09-17
Updated: 2026-09-17
Comments: Work in progress
License: http://creativecommons.org/licenses/by/4.0/
The gist: Information retrieval is increasingly important as LLM agents tackle complex tasks involving diverse information needs.
Terminology
Abstract
Information retrieval is increasingly important as LLM agents tackle complex tasks involving diverse information needs. Because retrieval relies on an index that represents each document through index keys, retrieval quality depends heavily on how effectively these keys expose the knowledge contained in each document. However, effective index representations vary across retrieval environments, making it difficult for any fixed optimization strategy to perform consistently. Yet evolving an index to its retrieval environment remains largely human-driven, requiring humans to diagnose retrieval failures, refine the optimization strategy, and reprocess the index accordingly. We propose SELF-INDEX, a framework that enables an index to self-evolve without human intervention. Its Optimizer autonomously diagnoses retrieval shortfalls, selectively revises the responsible index keys, and validates each revision before updating the index. Beyond reacting to observed retrieval demands, SELF-INDEX proactively explores additional demands through a Query Simulator, allowing the index to evolve beyond the queries already available for optimization. Across diverse corpora and retrievers, SELF-INDEX consistently improves retrieval performance while outperforming existing index optimization methods. We further show that these benefits extend to downstream applications, improving the effectiveness and efficiency of search agents and helping agent memory systems retrieve useful past interactions.
Sources
- BEAVER: An Enterprise Benchmark for Text-to-SQL
- Kimi K2.5: Visual Agentic Intelligence
- RL-Index: Reinforcement Learning for Retrieval Index Reasoning
- SPICE: Self-Play In Corpus Environments Improves Reasoning
- Document Expansion by Query Prediction
- AutoIndex: Learning Representation Programs for Retrieval
- gpt-oss-120b & gpt-oss-20b Model Card
- LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues
- Dr. Zero: Self-Evolving Search Agents without Training Data
- Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models
Related papers
- The Price of Isolation: Estimating the Ecosystem Cost of Symmetric Two-Sided A/B Testing
- SCAR: Semantic Continuity-Aware Retrieval for Efficient Context Expansion in RAG
- MixLoRA-DSI: Dynamically Expandable Mixture-of-LoRA Experts for Rehearsal-Free Generative Retrieval over Dynamic Corpora
- RRCM: Ranking-Driven Retrieval over Collaborative and Meta Memories for LLM Recommendation
- Right Family, Wrong Skill: Evaluating Risk Exposure in Agent Skill Retrieval
- UltRAG: a Universal Simple Scalable Recipe for Knowledge Graph RAG