KoViDoRe: A Benchmark for Korean Visual Document Retrieval

summary

The gist

The gist: KoViDoRe introduces a benchmark for Korean visual document retrieval that explicitly models multi-page evidence aggregation, showing that current models struggle with this complex task in

In short

KoViDoRe introduces a benchmark for Korean visual document retrieval that tests models on multi-page evidence aggregation, moving beyond single-page tasks. The benchmark reveals current models struggle with complex Korean layouts and structured content across multiple pages. A large training dataset, Ko-VDR Train Public, was also released to help develop better Korean visual document retrieval models.

Key concepts

KoViDoRe Benchmark: This is a new evaluation tool specifically designed for assessing how well AI models can retrieve information from complex Korean visual documents. Unlike previous benchmarks that only check single pages, KoViDoRe forces models to find evidence spread across many pages, simulating real-world document retrieval challenges.
Multi-stage Data Curation Pipeline: This is a detailed process used to create the KoViDoRe dataset. It involves several steps: parsing documents into structured data, creating diverse synthetic queries (both summary and context-based), and mapping relevance with human checks. This ensures the dataset accurately reflects real document complexity.
Ko-VDR Train Public: This is a large training dataset of 310,226 query-page pairs derived from Korean visual documents. It was created to help researchers fine-tune retrieval models specifically for Korean visual documents, showing that language-specific data significantly improves model performance.
Multi-hop Reasoning Queries: These are complex queries included in the dataset that require models to perform multi-step thinking to find an answer. They test a model's ability to aggregate information from different parts of a document or across multiple pages, which is crucial for handling complex Korean documents.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 18 episodes
Federated Learning — explained in 17 episodes
Quantum Fisher Information (QFI) — explained in 16 episodes
Retrieval-Augmented Generation (RAG) — explained in 16 episodes
Interpretability — explained in 15 episodes

This episode discusses

The paper

KoViDoRe: A Benchmark for Korean Visual Document Retrieval · Read on arXiv

Yongbin Choi, Yongwoo Song, Mujeen Sung

Kyung Hee University

Recent advances in multimodal retrieval have improved the ability to retrieve information from visually rich documents such as PDFs and reports. However, existing benchmarks remain largely centered on English and provide limited coverage of Korean visual documents with complex structures. Furthermore, most existing Korean resources primarily evaluate single-page retrieval, failing to capture realistic scenarios that require evidence aggregation across multiple pages. To address these gaps, we introduce KoViDoRe, a benchmark for Korean visual document retrieval. The dataset is constructed from publicly available Korean documents with diverse layouts, including tables, figures, and multi-column structures. We develop a multi-stage data curation pipeline consisting of structured document parsing, synthetic query generation using both summary-based and context-based strategies, and relevance mapping with human verification. Using KoViDoRe, we evaluate a wide range of multimodal retrieval models and observe that current models struggle to effectively handle Korean visual document retrieval, particularly in settings involving structured content and diverse query types. Motivated by this finding, we further curate a large-scale training dataset, Ko-VDR Train Public, to support the development of retrieval models tailored to Korean visual documents. Together, KoViDoRe and Ko-VDR Train Public provide a unified benchmark and training resource for Korean visual document retrieval.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "KoViDoRe: A Benchmark for Korean Visual Document Retrieval".

Jane: The gist: KoViDoRe introduces a benchmark for Korean visual document retrieval that explicitly models multi-page evidence aggregation, showing that current models struggle with this complex task in Korean settings,

Tom: First, who's behind it and why it matters.

Title and authors: Jane: The paper introduces "KoViDoRe: A Benchmark for Korean Visual Document Retrieval," and the authors are Yongbin Choi, Yongwoo Song, Mujeen Sung, all from Kyung Hee University.

Tom: They’re building this benchmark using publicly available Korean documents that have a really diverse set of layouts—things like tables and figures—which is key for testing real-world retrieval systems >

Lu: And they describe their data curation pipeline as a multistage process involving structured document parsing, synthetic query generation, and relevance mapping to make sure the data is high quality >

Meng: I'm interested in that query generation part; they use both summary-based and context-based strategies to create queries based on both the text and the visuals found during preprocessing >

Lalam: Plus they generate queries directly from local document context using page-level or local multi-page context windows, which should help capture that localized understanding of the document structure >

Tom: And they clean up everything with a relevance mapping step where they compare signals from the generation stage against a separate mapping stage to filter out inconsistent pairs >

The paper's summary: Jane: So what’s the core suggestion here is that we need to shift our focus away from single-page retrieval and towards multi-page evidence aggregation, which the authors see as a necessary change in how we evaluate these systems >

Tom: They argue that models should be able to identify a coherent set of pages instead of just pulling one relevant page, especially when you’re looking at things like detailed reports or manuals >

Lu: They point out that before this, most reasoning was confined to a single page or a very small context window, but this benchmark pushes the limits by demanding reasoning over multiple pages >

Meng: So the improvement they are highlighting is really testing if late-interaction models and dual-encoder approaches can actually handle this aggregation task effectively when dealing with Korean data >

Lalam: The paper notes that even though vision-language models are getting better at understanding text and visuals, the datasets they used for training and evaluation are still heavily concentrated in English and European languages >

Tom: So the main takeaway is addressing that lack of dedicated Korean benchmarks, which limits how we can actually test if retrieval models can work in realistic Korean document scenarios >

The paper's improvements: Jane: To summarize the improvements, the paper demonstrates that current multimodal retrieval models struggle with this multi-page evidence aggregation setting specifically in Korean contexts because they haven't been trained on language-specific data >

Tom: The main finding is that training on Korean-specific data can significantly improve performance, showing how important those language-specific supervision resources are for this type of task >

Lu: They also release a large-scale training dataset called KoVDR Train Public, which consists of over three hundred ten thousand query-page pairs derived from Korean visual documents >

Meng: That dataset is curated with filtering to remove low-quality queries, which is important because running models on noisy data at scale can be tricky when you’re shipping them to production >

Lalam: Fine-tuning two late-interaction retrieval models, colSmol and colqwen2 v1 point 0, on this KoVDR Train Public dataset consistently improved performance across both of them >

Tom: So the implication is that having Korean-specific training data combined with large-scale public training data generally yields the best overall performance for these Korean visual document retrieval tasks >

Conclusion: Jane: So to wrap up, KoViDoRe is this new benchmark for Korean visual document retrieval that really focuses on multi-page evidence aggregation, and it clearly shows that current models are struggling with that complexity >

Tom: It seems like the dataset construction pipeline they built was pretty detailed—they used structured document parsing to get page and element level markdown, then used both summary based and context based strategies for query generation >

Lu: I think the ambition there is really about how much reasoning these models can do across different sections of a document, not just looking at one isolated chunk of text >

Meng: From a deployment standpoint, having that dataset is useful because it gives us something concrete to train on, but I always wonder if that complex query generation pipeline translates cleanly to real traffic where you don't have perfect pre-processed structure >

Lalam: I see the point in their relevance mapping—comparing signals from the generation stage against a separate mapping stage to filter out inconsistent pairs and get better annotations >

Tom: And the results across those four domains, cybersecurity, energy, economic, and human resources show that performance gap between model scales is still there, but it also shows that even strong models hit a wall on certain subsets >

Jane: That’s what I found interesting—the economic and human resources domains consistently scored lower because of the complex document structures they mentioned earlier >

Lu: Yeah, the complexity of those structures seems to be a real bottleneck, indicating that just having more data isn't enough if it doesn't account for how information is actually distributed across pages >

Meng: For us, the implication is that we need models designed specifically to handle those multi-page aggregation tasks efficiently without getting bogged down in every single piece of evidence >

Lalam: And this brings us to KoVDR Train Public, which is that large-scale training dataset with over three hundred ten thousand query-page pairs they released specifically for fine-tuning late interaction models >

Tom: That fine-tuning showed consistent performance improvements on both colSmol and colqwen2 v1 point zero which suggests Korean specific supervision really helps the models learn the right retrieval patterns >

Jane: It’s a big step forward because it gives researchers a unified benchmark and a training resource for anyone trying to build retrieval systems for Korean visual documents >

Lu: The limitation they pointed out is that they rely on parsed markdown representations for query generation, which could introduce noise if the initial parsing isn't perfect >

Meng: That means future work needs to focus on integrating vision-language models directly into the query generation process to capture more of the visual information, not just what's already structured text >

Lalam: I think that’s where things are headed, moving beyond just structured text toward more integrated multimodal understanding in those Korean contexts >

Tom: So that’s KoViDoRe and KoVDR Train Public for now. Next up, we're going to look at how Ball Differential Privacy is trying to handle data reconstruction noise in other areas.

More episodes

← Home