Benchmarking Document Parsers on Mathematical Formula Extraction from PDFs

arXiv:2512.09874 · cs.CV, cs.AI, cs.IR · Submitted 2026-08-22 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Benchmarking Document Parsers on Mathematical Formula Extraction from PDFs".

Jane: The paper was written by Pius Horn and Janis Keuper from Institute for Machine Learning and Analytics (IMLA), University of Offenburg and University of Mannheim.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Summary: Tom: So, to recap, the researchers didn't just test existing documents; they built a synthetic framework with precise ground truth. This allows them to create a perfectly defined set of challenges for evaluation.

Jane: That’s a massive step because you can't really judge parsing quality if the target answer's representation is fuzzy or uncertain, so having that perfect LaTeX ground truth is essential for reliable testing and comparison.

Lu: I find the use of synthetic generation fascinating because it allows us to isolate exactly how a parser handles layout issues versus actual mathematical complexity, making the failure modes very clear.

Meng: And they didn't just test one system; they ran this against twenty different contemporary parsers, which is a really rigorous way to see where the weaknesses are in terms efficiency and accuracy.

Jane: They found significant performance disparities across all these tools, which makes the results immediately actionable for anyone building a knowledge base that relies on reliable parsing.

Tom: That’s right; we saw some top performers achieving scores over nine point six on average across those thousands of formulas they tested in their synthetic documents, showing where the current state-of-the-art is sitting.

Lalam: This means that instead of just hoping a PDF parser works, we now have a solid, scientifically validated way to measure its success, which is huge for establishing trust in AI systems that use academic content.

Meng: It also helps us understand that the quality of the output isn' directly tied to the specific tools we choose and how well they handle complex structure and formatting within their processing pipeline.

Lu: I think it suggests a shift towards prioritizing semantic accuracy over simply trying to get a perfect score on evaluation, which is a very modern approach to assessment that goes beyond simple string matching.

Tom: That leads us perfectly into the core of the methodology in "Benchmarking Document Parsers on Mathematical Formula Extraction from PDFs" and how they measure success despite these structural challenges.

Improvements: Tom: The biggest hurdle, as the paper outlines, is that traditional methods for checking if a parsed formula is correct—like basic string matching—just break down because of representation ambiguity.

Jane: Because mathematical concepts aren't unique; there are dozens of ways to write one equation while keeping its meaning the same, so simple metrics like Levenshtein distance can't tell the difference between a true representation and a minor typo.

Lu: The introduction of LLM-as-a-judge is such a brilliant solution here because it allows us to assess the *meaning* of the formula, not just its surface syntax or its visual appearance.

Meng: It’s an engineering breakthrough that turns an unstable problem—where parser output varies wildly—into a stable system by using the LLM's semantic understanding for matching and validation.

Jane: The two-stage matching pipeline is key here; it combines that powerful LLM logic with deterministic fuzzy validation to handle all the format inconsistencies we see in real data.

Tom: That means even if a parser omits delimiters or messes up the sequence in a multi-column layout, this system can still find the correct match using semantic understanding of context.

Lalam: This is how we move beyond just looking at symbols and start valuing the actual knowledge contained within those equations, which is a huge step for cultural preservation of scientific thought.

Meng: It’ provides us with much more robust data than before, allowing us to train LLMs on scientific content with far greater confidence in the underlying accuracy of the training material.

Lu: By showing that an LLM correlation of zero point seven eight works so well against human judgment, it gives us a highly reliable metric that we can trust for future validation efforts to measure performance.

Tom: It’s clear that "Benchmarking Document Parsers on Mathematical Formula Extraction from PDFs" is providing us with the tools to measure success in a way science demands, and we're ready to move into the results section.

Evaluation & Results Nuance: Tom: We've seen how this paper tackles the fundamental difficulty of PDF parsing by creating a robust, scientifically controlled environment for mathematical evaluation. It’s hard to imagine doing any scientific research without accurate data extraction.

Jane: It’s genuinely thrilling because we have moved past simple string comparison and embraced a method that understands what is actually meant by using LLMs to check semantic equivalence instead of just checking character matches.

Lu: I'm excited about how this opens up pathways for complex scientific research, allowing us to finally unlock the true potential of our academic archives, which are often locked behind these parsing challenges.

Meng: From an engineering standpoint, we now have a clear roadmap for selecting and optimizing parsers based on real-world performance under pressure, rather than just picking the cheapest or most accessible tool.

Lalam: The paper has helped us understand that the future of knowledge acquisition is not just about volume of data, but about having semantic fidelity in how we process and store that information.

Tom: This leads directly to why certain parsers are performing so much better than others, which is shown by the leaderboard in "Benchmarking Document Parsers on Mathematical Formula Extraction from PDFs."

Jane: It’s interesting because while tools like Character Detection Matching or CDM might catch most errors, they aren't good at detecting when a semantically different symbol that looks similar to a human eye are those subtle mistakes.

Lu: I think the LLM approach captures these subtle semantic errors much more effectively than traditional metrics, which is where the real value of this new evaluation lies.

Meng: The data shows that specialized tools like Mathpix and dots.ocr have significantly higher scores compared to general-purpose tools, indicating that we need purpose-built systems for high-stakes document parsing.

Lalam: This shows us that trust in AI is tied directly to the tool selection, and the right a dedicated parser can be far more impactful than generalized intelligence.

Tom: The performance gap is undeniable, and it’s important for practitioners to understand why choosing a specific parser is so critical for downstream tasks.

Conclusion: Tom: So, that really wraps up our deep dive into "Benchmarking Document Parsers on Mathematical Formula Extraction from PDFs." It seems like this area is getting incredibly complex because math formulas introduce such a high level of structural difficulty compared to just plain text.

Jane: You're right, Tom. What struck me most was how much specialized effort is needed just to understand the *structure* of an equation, not just the characters themselves. It’s like going from reading a sentence to reading sheet music—you need a a whole different set of rules.

Lu: I think the implications here go way beyond just academic parsing, though. When you can reliably extract and structure math equations from diverse PDF formats, you're opening up huge possibilities for automated scientific literature review and drug discovery pipelines.

Meng: From an engineering standpoint, Lu has a point about pipelines. The biggest hurdle I see is standardization—if every document source is slightly different in how it formats its math, the cost of building a robust parser becomes astronomical.

Lalam: And that variability is precisely what advanced AI models are being trained to handle. The ability to generalize across these complex formats means we're moving closer to truly universal knowledge extraction, improving how cultures share their deepest ideas.

Tom: Meng brought up a good point about standardization, Jane. It makes me wonder if the future of this research lies in creating a universal intermediate representation for math formulas that all parsers can target.

Jane: Exactly! Because right now, every parser spits out something slightly different, even if they're extracting the same formula from the same document type.

Lu: If we could achieve that unified output format, it wouldn't just be a better tool; it would be an infrastructural leap for scholarly publishing itself.

Meng: And that kind of reliable infrastructure is what companies need to build commercial products around, frankly. Knowing your input quality is consistent changes everything about development timelines.

Lalam: It elevates the entire digital reading experience, making complex knowledge accessible to people who don't have the background to manually process all those equations and tables.

Tom: Okay, this has been incredibly insightful for us listeners; we really appreciate you walking us through this challenging topic. We’ll definitely be following developments in document parsing.

Lu: I'm ready to see what possibilities await the next paper, especially regarding how these structural AI advances will redefine how we learn and work with complex knowledge.

Meng: I'm already thinking about how to implement these new benchmarks in my systems to ensure accuracy at scale.

Lalam: The pursuit of structured knowledge is fundamental to human progress, and this research just amplifies that potential for the next generation, making a more informed future possible.

Pius Horn, Janis Keuper

Institute for Machine Learning and Analytics (IMLA), University of Offenburg · University of Mannheim

cs.CV, cs.AI, cs.IR

Submitted: 2026-08-22

Updated: 2026-08-25

Code: https://github.com/phorn1/pdf-parse-bench

Importance score: 76/100

The gist: Correctly parsing mathematical formulas from PDFs is critical for training large language models and building scientific knowledge bases from academic literature, yet existing benchmarks either

Key concepts

Synthetic Framework: The researchers created a perfectly defined set of challenges by building a synthetic framework with precise ground truth. This allows them to create a controlled environment where the target answer is known, which is essential for reliable testing and comparison of parsing quality.
Semantic Validation: Because mathematical concepts can be written in dozens of ways while maintaining the same meaning, simple string matching fails. The introduction of LLM-as-a-judge allows assessment based on the formula's actual meaning, not just its visual appearance or surface syntax.
Parser Benchmarking: This involved running tests against twenty different contemporary parsers to rigorously evaluate their performance. The goal is to identify weaknesses in efficiency and accuracy, providing a scientifically validated way to measure success in selecting document parsing tools.

Terminology

Benchmark — 31× in this paper · explained in 2 episodes
Benchmarking — 20× in this paper · explained in 3 episodes
LLM-as-a-Judge — 12× in this paper · explained in 3 episodes
Accuracy — 6× in this paper · explained in 1 episode
Association for Computational Linguistics — 3× in this paper
International Workshop on Document Analysis Systems — 3× in this paper
Semantic Understanding — 3× in this paper · explained in 2 episodes
Character Detection Matching — 2× in this paper
Computer Vision and Pattern Recognition — 2× in this paper
Core — 2× in this paper · explained in 2 episodes
Consistency — 1× in this paper · explained in 3 episodes
Downstream Tasks — 1× in this paper · explained in 2 episodes
Failure Modes — 1× in this paper · explained in 3 episodes
Fidelity — 1× in this paper · explained in 2 episodes
International Conference on Document Analysis and Recognition — 1× in this paper
Knowledge Base Construction — 1× in this paper · explained in 2 episodes
LLM judge — 1× in this paper · explained in 2 episodes
Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
RAG — 1× in this paper · explained in 3 episodes
Vision-Language Models (VLMs) — 1× in this paper · explained in 6 episodes

Summary

Correctly parsing mathematical formulas from PDFs is critical for training large language models and building scientific knowledge bases from academic literature, yet existing benchmarks either exclude formulas entirely or lack semantically-aware evaluation metrics. To address this gap, we introduce a benchmarking framework centered on synthetically generated PDFs with precise LaTeX ground truth, enabling systematic control over layout, formulas, and content characteristics.

The difficulty in PDF parsing stems from the format's design for visual presentation rather than semantic content representation. Mathematical formulas are particularly challenging due to their extensive symbol sets, two-dimensional structure where spatial positioning conveys meaning (superscripts, subscripts, fractions), and the need to convert visual arrangements into structured formats like LaTeX.

Our methodology involves three main components: (1) a formula dataset extracted from Wikipedia (the wikipedia-latex-formulas-319k collection), (2) synthetic benchmark PDFs generated with precise ground truth by randomly combining sampled formulas... with text segments and inline formulas using randomly composed LaTeX templates, and (3) an evaluation pipeline that matches and assesses parsed formulas.

A major challenge in formula matching is the substantial variations in parser output formats. To overcome this, we developed a robust two-stage matching strategy:

LLM-Based Formula Extraction: We employ GPT-5-mini to extract formulas from the parsed markdown, which provides reliable extraction quality at modest computational cost.
Fuzzy Matching Validation: We validate each extraction using deterministic fuzzy matching—after attempting exact substring matching, we normalize both strings by removing whitespace and backslashes, then use sliding window Levenshtein distance to locate the best match in the original text.

We evaluate formula quality using LLM-as-a-judge for semantic assessment. We validated this approach through a human study involving 250 formula pairs (750 ratings from 30 evaluators), demonstrating a Pearson correlation of r=0.78 with human judgment, compared to r=0.34 for character-level matching (CDM) and r≈0 for text similarity.

The results of evaluating over 20+ contemporary PDF parsers across 100 synthetic documents containing more than 2,000 formulas reveal significant performance disparities, providing actionable guidance for practitioners selecting parsers for downstream applications. This framework establishes a semantically-aware, reproducible evaluation methodology for mathematical content extraction from PDFs.

Improvements for AI systems

As a fastidious and highly rigorous AI researcher, I have analyzed this paper to extract specific, actionable improvements that address fundamental weaknesses in current PDF processing pipelines. The core of the paper is not just a benchmark; it is a methodology for semantic-aware document understanding.

The following improvements should be integrated into any high-stakes AI system (e.g., RAG systems, scientific knowledge base builders, LLM training corpora) that processes academic PDFs containing mathematical content.

Current Weakness Addressed: Relying on manually annotated or scraped data often results in inconsistent ground truth and fails to control for specific layout/formatting challenges (e.g, merged formulas, delimiters).

The Improvement: Integrate a controlled, synthetic PDF generation framework into the training and validation datasets.

Actionable Implementation:

Create a Diverse Formula Corpus: Utilize a curated collection of complex mathematical expressions (like the wikipedia-latex-formulas-319k dataset) as the source of all atomic content.
Systematically Generate Test Cases: Build synthetic PDF documents by randomly combining these formulas with text segments, and iteratively compiling them using a controlled LaTeX template. This allows for precise control over:

Layout Variability: Testing single-column vs. two-column layouts, font families, and spacing constraints.
Content Density: Generating documents that are densely filled to simulate real academic papers without relying on natural document structure biases.

What the Improved System Can Do: The system will have a robust, reproducible test suite that goes beyond simple content matching, ensuring it can handle complex structural transformations and synthetically induced parsing errors.

Current Weakness Addressed: Traditional rule-based parsers (e.g., those using regex or fixed delimiters) fail catastrophically when parsers omit formulas, merge them, or introduce inconsistent formatting (e.g., missing, delimiters).

The Improvement: Replace rigid, deterministic matching with a hybrid pipeline that integrates semantic understanding before applying deterministic validation.

Actionable Implementation:

Stage 1: Semantic Extraction (LLM-Guided): Use a specialized LLM (e.g., GPT-5-mini) to extract formulas from the parsed raw text/markdown output. The LLM must be instructed to maintain the original sequence and identify potential grouping issues, returning a structured JSON output listing all extracted elements.
Stage 2: Fuzzy Validation (Deterministic Check): For every formula identified in Stage 1, perform a deterministic check against the precise ground truth:

Normalize both strings (remove whitespace, normalize backslashes).
Calculate the Levenshtein distance using a sliding window approach.
Accept the the extraction if it falls within a defined edit distance threshold of the original ground truth.

Retry Mechanism: If validation fails, employ a focused retry mechanism to isolate and re-evaluate only use unparsed or ambiguous segments, preventing cascading errors.

Current Weakness Addressed: Traditional metrics (BLEU, Levenshtein Distance, Character Detection Matching [CDM]) are fundamentally inadequate because they penalize structural differences or character errors uniformly, failing to distinguish between a minor typographical error and a fundamental semantic change.

The Improvement: Establish LLM-based semantic equivalence as the primary gold standard for measuring parsing quality.

Actionable Implementation:

Define Semantic Criteria: Program the evaluator (LLM) to score formula pairs (Ground Truth vs. Parsed Output) on a 0–10 scale based on three specific criteria: Correctness, Completeness, and Semantic Equivalence.
Weight Semantic Fidelity: Prioritize the LLM's assessment of semantic equivalence over character-level matching. The LLM must be instructed to recognize that syntactically different but mathematically identical formulas (e.g., 1 over 2 vs 12) receive a high score, while formulas with incorrect mathematical meaning receive a low score.
Calibration: Use the human study results (r=0.78 correlation) to calibrate the LLM's scoring thresholds, ensuring its automated judgments align closely with human expert consensus.