RATIO: A Benchmark for Retrieval Across Typed Ideation Operations in Scientific Literature
cs.CL, cs.IR
Submitted: 2026-08-27
Updated: 2026-09-01
License: http://creativecommons.org/licenses/by/4.0/
The gist: Retrieved scientific literature can serve as inspiration for both human and AI scientists.
Terminology
Abstract
Retrieved scientific literature can serve as inspiration for both human and AI scientists. Inspiration can take different forms: prior work may directly suggest how to address a problem, or surface directions at different levels of abstraction - zooming out to a more general view or zooming in to a concrete realization. We introduce RATIO (Retrieval Across Typed Ideation Operations), a large-scale benchmark in which relevance is defined by three operations which we name ideation moves: Address retrieves potential approaches for stated problems, Broaden retrieves more general formulations, and Specify retrieves concrete instantiations. RATIO is constructed from millions of full-text scientific papers across CS literature via a general recipe that extends discourse-marker distant supervision - previously used only for classification - to corpus-scale retrieval, combined with extensive LLM and human vetting. Experiments show that operation-specific fine-tuning substantially boosts retrievers but leaves much room for further improvements. RATIO provides a scalable training and evaluation framework for retrieval components that support literature-grounded ideation, opening up new research avenues on scientific inspiration retrieval.
Sources
- Creating and controlling global Greenberger-Horne-Zeilinger entanglement on quantum processors
- Efficient Natural Language Response Suggestion for Smart Reply
- The Semantic Scholar Open Data Platform
- BM25S: Orders of magnitude faster lexical search via eager sparse scoring
- Representation Learning with Contrastive Predictive Coding
- Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference
- Jasper and Stella: distillation of SOTA embedding models
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering