SHELF: A Synthetic Harness for Multi-Task Bibliographic Benchmarking

arXiv:2609.03047 · cs.CL, cs.AI, cs.IR · Submitted 2026-09-02 · Read on arXiv

cs.CL, cs.AI, cs.IR

Submitted: 2026-09-02

Updated: 2026-09-02

Code: https://github.com/mjbommar/shelf-benchmark

Terminology

• Benchmark — 46× in this paper · explained in 2 episodes
• Clustering — 24× in this paper · explained in 2 episodes
• Accuracy — 14× in this paper · explained in 1 episode
• Embeddings — 10× in this paper · explained in 2 episodes
• AUC — 6× in this paper · explained in 2 episodes
• Benchmarking — 6× in this paper · explained in 3 episodes
• Core — 5× in this paper · explained in 2 episodes
• Cosine Similarity — 3× in this paper · explained in 3 episodes
• Provenance — 3× in this paper · explained in 4 episodes
• External Validation — 2× in this paper · explained in 2 episodes
• Semantic Similarity — 2× in this paper · explained in 2 episodes
• Transformer — 2× in this paper · explained in 2 episodes
• Calibration — 1× in this paper · explained in 7 episodes
• Distillation — 1× in this paper · explained in 2 episodes
• Grounding — 1× in this paper · explained in 3 episodes
• Knowledge Graph — 1× in this paper · explained in 7 episodes
• Linear probe — 1× in this paper · explained in 2 episodes
• Massive Text Embedding Benchmark — 1× in this paper
• Self-Supervised Learning — 1× in this paper · explained in 8 episodes

Sources

• OpenGloss: A Synthetic Encyclopedic Dictionary and Semantic Knowledge Graph
• Neural Retrievers are Biased Towards LLM-Generated Content
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• Grounding Synthetic Data Evaluations of Language Models in Unsupervised Document Corpora
• The Embedder's Dilemma: LLMs Are Better, but at What Cost?
• AI and the Everything in the Whole Wide World Benchmark
• Text Embeddings by Weakly-Supervised Contrastive Pre-training
• DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
• MPNet: Masked and Permuted Pre-training for Language Understanding
• LCSHBench: A Multilingual, Consensus-Grounded Benchmark for Library of Congress Subject Heading Assignment
• C-Pack: Packed Resources For General Chinese Embeddings
• Rethinking Benchmark and Contamination for Language Models with Rephrased Samples
• EmbeddingGemma: Powerful and Lightweight Text Representations
• A Careful Examination of Large Language Model Performance on Grade School Arithmetic

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering