Document-as-Image Representations Fall Short for Scientific Retrieval
summary
The gist
The gist: document-as-image representations are consistently suboptimal for scientific retrieval, especially in dense documents where evidence is distributed across structured sources like text,
In short
Current document embedding models rely on treating documents as images of rendered pages, which is inefficient for complex scientific texts containing text, tables, and figures. The authors introduce ArXivDoc, a new benchmark built from raw LaTeX sources. Comparing text-only, image-based, and multimodal representations shows that interleaved text+image methods outperform image-based approaches without extra training.
Key concepts
- Document-as-Image Representations
- These models embed documents by treating them like photographs of a rendered page. This approach is often used but fails in scientific contexts because critical evidence is spread across structured elements like tables and equations, which are lost when only the visual image is considered.
- ArXivDoc Benchmark
- This new benchmark uses the original LaTeX source code of scientific papers instead of PDFs or images. This allows for controlled queries targeting specific structured elements (like sections or figures), providing a richer dataset for testing different document representations.
- Multimodal Representations
- These are embedding methods that combine different data types, such as text and images, into a single vector representation. The study finds that combining text and image information effectively is better than using just one type of representation alone for scientific retrieval.
Terminology used across episodes
This episode discusses
- Document-as-Image Representations Fall Short for Scientific Retrieval · Paper Radio
- Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Tasks
- Glyph: Scaling Context Windows via Visual-Text Compression
- PaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Model
- MDocAgent: A Multi-Modal Multi-Agent Framework for Document Understanding
- E5-V: Universal Embeddings with Multimodal Large Language Models
- Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking
- BM25S: Orders of magnitude faster lexical search via eager sparse scoring
- ViDoRe Benchmark V2: Raising the Bar for Visual Retrieval
- Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
- REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmark
- DeepSeek-OCR: Contexts Optical Compression
- VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents
- MRMR: A Realistic and Expert-Level Multidisciplinary Benchmark for Reasoning-Intensive Multimodal Retrieval
- Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models
- Efficient generation and control of different order orbital angular momentum states for communication links
- Crossover phenomena involving the dense O(n) phase
- The Infrared Emission and Opening Angle of the Torus in Quasars
- Micromotion-induced Limit to Atom-Ion Sympathetic Cooling in Paul Traps
- Locally exact modifications of numerical schemes
- Searching for tidal tails around omega Centauri using RR Lyrae Stars
The paper
Document-as-Image Representations Fall Short for Scientific Retrieval · Read on arXiv
Ghazal Khalighinejad, Raghuveer Thirukovalluru, Alexander H. Oh, Bhuwan Dhingra
Duke University
Many recent document embedding models are trained on document-as-image representations, embedding rendered pages as images rather than the underlying source. Meanwhile, existing benchmarks for scientific document retrieval, such as ArXivQA and ViDoRe, treat documents as images of pages, implicitly favoring such representations. In this work, we argue that this paradigm is not well-suited for text-rich multimodal scientific documents, where critical evidence is distributed across structured sources, including text, tables, and figures. To study this setting, we introduce ArXivDoc, a new benchmark constructed from the underlying LaTeX sources of scientific papers. Unlike PDF or image-based representations, LaTeX provides direct access to structured elements (e.g., sections, tables, figures, equations), enabling controlled query construction grounded in specific evidence types. We systematically compare text-only, image-based, and multimodal representations across both single-vector and multi-vector retrieval models. Our results show that: (1) document-as-image representations are consistently suboptimal, especially as document length increases; (2) text-based representations are most effective, even for figure-based queries, by leveraging captions and surrounding context; and (3) interleaved text+image representations outperform document-as-image approaches without requiring specialized training.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: Today's paper: "Document-as-Image Representations Fall Short for Scientific Retrieval".
Jane: The gist: document-as-image representations are consistently suboptimal for scientific retrieval, especially in dense documents where evidence is distributed across structured sources like text, tables, and figures.
Tom: First, who's behind it and why it matters.
Paper summary: Tom: So wrapping up "Document-as-Image Representations Fall Short for Scientific Retrieval", the paper is basically arguing that we need to move away from just embedding rendered pages as images for scientific searching. They specifically introduce ArXivDoc built on LaTeX sources to test this, and their main finding is that document-as-image representations are consistently suboptimal, especially with longer documents.
Jane: It really boils down to the fact that critical evidence in science papers isn't just in the rendered image; it's distributed across text and tables and figures, so a representation that treats it all as one image just misses most of the signal.
Lu: The authors stress that LaTeX provides direct access to those structured elements like sections tables figures and equations which allows for controlled query construction based on specific evidence types. This shifts the focus from just looking at the page layout to understanding the underlying structure.
Meng: From a production standpoint, this suggests that if you're building an AI system for scientific search, you can't just rely on a single image embedding. You need something that understands how text and figures relate to each other structurally.
Lalam: The paper's core contribution is showing that interleaved text image representations outperform document-as-image approaches without requiring any additional training, which is a significant finding because it suggests combining modalities is inherently more robust than relying on just rendered pages.
Tom: So the implication for us listening right now is that if you are working on scientific retrieval, you should probably stop thinking of documents as just images and start thinking about how text and those other structured elements work together.
Jane: That's right, because the authors suggest extending ArXivDoc to support queries that require combining evidence across multiple parts of a document or even across documents can really advance retrieval in these scientific settings.
Conclusion: Tom: So we’re wrapping up this talk on "Document-as-Image Representations Fall Short for Scientific Retrieval." Basically, the paper is showing that treating scientific documents just as images of a page isn't working well anymore.
Jane: Right, so if you think about it, most of the big models out there were trained on those images, right? But this study builds a new benchmark called ArXivDoc using actual LaTeX source code.
Lu: That’s the core idea—LaTeX lets you see the structure directly, like where the tables and figures are placed, instead of just looking at how it looks on a screen.
Meng: And what they found is that when you use those image-based representations, performance gets worse as the document gets longer. That's a real concern for any system trying to handle long papers.
Lalam: The paper shows that text-only representations actually do surprisingly well for queries about figures too, because the text around the figure gives away what’s in it.
Tom: I’m interested in how they compare those different ways of representing the data, especially seeing how interleaved text and image stuff performs without needing any special extra training.
Jane: That seems pretty important for us trying to build systems that actually understand scientific evidence, not just the visual layout.
Lu: It points out a gap between how these complex scientific documents are actually built in LaTeX and how the current retrieval models were designed to learn from images.
Meng: If we’re deploying something at scale, we need to know what breaks when you move from that training data to real-world traffic, which this benchmark helps us test.
Lalam: The authors suggest that maybe training models directly on these interleaved text and image documents is a promising way forward for getting better results.
Tom: So the authors are essentially saying the current standard approach of document-as-image is consistently suboptimal, and they’re suggesting we look at structured sources like LaTeX as a better way to build those embeddings.
Jane: Exactly. It’s about moving beyond just seeing a picture of a page and starting to see how text and structure actually work together in science.
Tom: And that opens up the idea of building more effective representations for these dense, multimodal scientific papers across the board.
More episodes
- 2604.03925-AdaptFuse: Training-Free Sequential Preference Inference via Externalized Bayesian Computation
- 2604.18344-DiffTSP: A Conditional Discrete Diffusion Model for Knowledge Graph Triple Set Prediction
- 2512.13399-Dual-Evolving Reinforcement Learning for Automated Reward Design
- 2512.22174-BitFlipScope: Scalable Fault Localization and Recovery for Bit-Flip Corruptions in LLMs
- 2601.07939-SECite: Analyzing and Summarizing Citations in Software Engineering Literature
- 2605.10337-CORTEG: Foundation Models Enable Cross-Modality Representation Transfer from Scalp to Intracranial Brain Recordings
- 2601.22449-Emergence of Physical Intelligence via Controllable Information Production
- 2602.01779-LingLanMedBench: A Unified Benchmark for Evaluating Large Language Models in Traditional Chinese Medicine
- 2602.08353-TKGB: A Comprehensive Benchmark for Temporal Knowledge Graph Forecasting
- 2603.00578-Draft-Thinking: Learning Efficient Reasoning in Long Chain-of-Thought LLMs