Optimal Transport for Handwritten Text Recognition in a Low-Resource Regime

arXiv:2509.16977 · cs.CV, cs.LG · Submitted 2026-08-22 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Optimal Transport for Handwritten Text Recognition in a Low-Resource Regime".

Jane: The paper was written by Petros Georgoulas Wraight, Giorgos Sfikas, Ioannis Kordonis, Petros Maragos and George Retsinas from Robotics Institute, Athena Research Center, Maroussi, Greece and National Technical University of Athens, School of ECE, Greece and University of West Attica, Department of SGE, Athens, Greece and HERON - Hellenic Robotics Center of Excellence.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Summary of Approach: Tom: So, we know we're tackling a low-resource environment, but how does this specific framework manage that scarcity? Jane, can you give us the high-level summary of their approach without getting lost in the math?

Jane: Essentially, they are treating handwritten text recognition not as a simple image classifier, but as a visual-semantic matching task. They start with just a tiny handful of labeled examples—a minimal set—and then they iteratively teach the system how to use those limited examples to find similar images in the unlabeled data.

Meng: That iterative bootstrapping process is key, which is where the magic happens. They take these minimal labeled points and use them to generate pseudo-labels for high-confidence matches, allowing us to expand our training set exponentially without adding more human effort.

Lu: The concept of leveraging a "lexical prior" within that summary is also fascinating. By using knowledge about how words are structured and how often they appear, the system gains a baseline understanding of language that guides where it should be looking in the visual space.

Lalam: It’s like giving the AI a linguistic intuition before it even sees the data. This allows it to make smart guesses about what an image *should* be based on common vocabulary, making its eventual "guesses" much more reliable than random chance.

Tom: That reliance on prior knowledge is a huge advantage in rare documents where context is everything. Jane, how does this iterative process actually manage the transition from these initial seeds to a larger training set?

Jane: It’s not just guessing; the system is actively aligning visual features with semantic word representations using Optimal Transport. This sophisticated alignment allows us to identify which unlabeled image corresponds most closely to a specific word in the lexicon.

Meng: The iterative nature of this process means that as we add more and more of these high-confidence pseudo-labels, the entire network gets retrained and refined, making it a self-improving system rather than a static model.

Lalam: It ensures that even when dealing with highly unique handwriting styles—styles we've seen only once or twice before—the system can still place that word in its proper context within the language structure.

Tom: That’s a powerful combination of self-improvement and linguistic grounding. Now, let's transition to how these conceptual improvements translate into actual performance gains by looking at the results section.

Improvements and Results: Tom: We’ve looked at the core idea—the iterative alignment process—but what does the data actually tell us about how much better this system is than previous methods? Jane, what are the most impressive quantitative findings from "Optimal Transport for Handwritten Text Recognition in a Low-Resource Regime"?

Jane: The results show that this model significantly improves recognition accuracy across various benchmarks like GW and CVL, achieving improvements that exceed ten percent over the current state of the art.

Lu: I think it’s crucial to understand *where* those improvements are most noticeable: they are particularly dramatic in the low-label regimes—when we only have a tiny fraction of labeled data.

Meng: From an engineering perspective, this means that if we' can deploy this on real-world archives with limited human resources, the ROI is incredibly high because the system is so efficient at learning from minimal input.

Lalam: For us handling diverse and rare scripts, this robust performance suggests that the limitations of resource constraints aren't a dead end; it means we can accurately transcribe materials previously considered too difficult to process.

Tom: That’s huge for accessibility. The results also mention ablation studies where they test different parts of the model. Jane, did those tests show what specific component contributed most to this success?

Jane: Yes, the comparison shows that including the PHOC auxiliary head is vital; without it, performance drops considerably, proving that this specific linguistic regularizer helps stabilize and boost accuracy across all tested datasets.

Meng: It’s not just one piece of technology; it's the combination of these components—the core OT alignment plus the PHOC head—that makes the system reliable enough to be deployed in a real-world, high-stakes archival setting.

Lu: The fact that we see performance across different datasets like GW and CVL confirms that this isn' not just a localized fluke but a generally applicable methodology for finding semantic structure.

Lalam: It ensures that the potential of our cultural data is fully realized, allowing us to interpret those historical records with confidence in their underlying meaning.

Tom: We have seen how much better it performs and why, but now we need to dig into the technical heart: how does this actual math translate into a readable transcription? Let's break down the process.

Core Mechanism Breakdown: Tom: We’ve discussed *why* this system works and *how much* it improves, but now let's look at the "how" in the core mechanism of "Optimal Transport for Handwritten Text Recognition in a Low-Resource Regime." Jane, can you give us a simple step-by-step breakdown of how a single image is transformed into a word?

Jane: Think of it as transforming the visual characteristics of an image into two distinct spaces: first, we have the visual space where the image sits, and second, we have the semantic space where all possible words live. The system then uses Optimal Transport to find the most probable match between those two spaces.

Lu: That's where it becomes really elegant; instead of just looking for visual similarities at one step, we are mapping entire distributions. We are finding a path of minimum cost that connects the image's features to the vocabulary embeddings, preserving the relationship between words.

Meng: This process is sophisticated because it involves two distinct training phases—the initial backbone adaptation and then training a lightweight projector. The projector takes that holistic visual descriptor and maps it into that specific word embedding space for alignment.

Lalam: This creates an incredibly robust system because we are not just guessing the next letter; we are mapping the entire context of a word against its semantic meaning, which is vital for understanding how language flows across time and space.

Tom: So, we're moving from pixel patterns to semantic probability. Jane, how does the "pseudo-label expansion" fit into this highly technical process?

Jane: The system uses the transport plan provided by the OT solver—that pathfinding mechanism—as confidence scores for every possible word in a lexicon. If the mass is tightly concentrated on one word, that image gets a high confidence score and becomes a pseudo-label.

Meng: We then select top candidates based on low Shannon entropy, which means we are selecting images where the system is most certain of its own judgment, moving those high-confidence items into the training set for further refinement.

Lu: By using OT to guide our selection, we ensure that we aren't just taking random samples; we are strategically picking samples that represent the most linguistically probable alignment between a visual input and a known vocabulary entry.

Lalam: This ensures that even in rare or unique handwriting styles, the system is guided by language structure rather than pure chance, allowing us to see meaning where previously there was only visual noise.

Tom: It sounds like we have seen both the "what" and now the detailed "how." We’ve covered everything from the initial concept to its core mechanism. Now for our final wrap-up segment.

Conclusion and Wrap-Up: Tom: We've explored all facets of "Optimal Transport for Handwritten Text Recognition in a Low-Resource Regime," from the initial idea to its mechanical execution, and it's clear this is a massive leap forward. Jane, do you feel we have captured the full scope of what makes this paper so significant?

Jane: It truly shows that we don’t need millions of labeled examples anymore; that this system can be highly effective even with very limited data, which makes it incredibly practical for researchers dealing with small or rare collections.

Lu: The sheer flexibility provided by the optimal transport framework allows us to see a potential future where we map complex sequence understanding far beyond just text—the mathematical possibilities are genuinely exciting.

Meng: From an engineering perspective, the fact that this method is so robust means we can build reliable systems that work consistently across multiple real-world scenarios without needing an exhaustive, human-powered labeling effort.

Lalam: We are using the principles of "Optimal Transport for Handwritten Text Recognition in a Low-Resource Regime" to ensure that ancient and often overlooked cultural heritage can be accurately transcribed and understood by both current researchers and future generations.

Tom: It's a perfect blend of sophisticated math, practical engineering, and profound cultural impact.

Lu: I think it confirms that structural intelligence, not just raw data volume, is what drives the most effective AI systems.

Meng: And confidence in accuracy is a huge win for us when we have to deploy these tools on real-world archives where failure's simply isn't an option.

Lalam: It’s about ensuring the stories remain accessible because of this technique that allows us to see the potential in every single piece of paper.

Tom: We have a lot to unpack from this breakthrough, but for now, let's take a quick break and come back with another exciting paper on arXiv.

Petros Georgoulas Wraight, Giorgos Sfikas, Ioannis Kordonis, Petros Maragos, George Retsinas

Robotics Institute, Athena Research Center, Maroussi, Greece · National Technical University of Athens, School of ECE, Greece · University of West Attica, Department of SGE, Athens, Greece · HERON - Hellenic Robotics Center of Excellence

cs.CV, cs.LG

Submitted: 2026-08-22

Updated: 2026-08-25

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 56/100

The gist: " However, traditional HTR methods rely on "extensive annotated sets for training," which makes them "impractical for low-resource domains like historical archives or limited-size modern

Key concepts

Optimal Transport
This is a mathematical tool used to find the most probable path of minimum cost between two distributions. In this context, it maps visual image features to semantic word representations to find the best possible match.
Iterative Bootstrapping
This process involves starting with a small set of labeled examples and using them to generate pseudo-labels for unlabeled data. This allows the system to expand its training set exponentially without needing more human labeling effort.
Lexical Prior
This refers to using existing knowledge about language structure, such as how words are typically organized and how often they appear. It gives the AI a baseline understanding of language to guide its visual search for matches.

Terminology

Summary

The following is a detailed summary of the scientific paper, quoting relevant sections of the text:

Handwritten Text Recognition (HTR) is a task of central importance in the field of document image understanding. However, traditional HTR methods rely on extensive annotated sets for training, which makes them impractical for low-resource domains like historical archives or limited-size modern collections. This reliance on supervised learning is particularly difficult in low-resource environments where labeled data are scarce or expensive to produce.

The authors aim to overcome these limitations by reframing HTR not as a standard supervised task, but as visual–semantic matching and proposing an iterative bootstrapping framework that aligns visual descriptors of unlabeled word images with semantic word representations using Optimal Transport (OT).

The paper's key novelty lies in its ability to leverage a lexical prior, which is knowledge about valid word instances and relative frequencies in the target vocabulary. This provides a distribution over the semantic space onto which images from the visual space are to be aligned.

The contribution of this work is twofold:

  1. We introduce a novel model paradigm for HTR, that casts the problem under an intuitive Optimal Transport-based self-training scheme.

  2. The model can leverage lexical prior knowledge to obtain a considerable boost in performance (up up to more than 10% of improvement over the current state of the art).

The framework maintains two complementary spaces: a visual space for word-image descriptors and a lexical embedding space for candidate tokens.

  • Visual Space: The network uses a compact residual CNN (the backbone). An input image I is mapped to a sequence of column descriptors C. This sequence feeds into two branches:

  1. Sequential Transcription Head: A bidirectional GRU processes the descriptors to produce states, which are then mapped by a linear layer to logits w over the alphabet. This provides frame-wise logits trained with the Connectionist Temporal Classification (CTC) loss, used for lexicon-free decoding at inference.

  2. Holistic Descriptor: The recurrent states are averaged and projected to form a global image vector z, which serves as a single point in the the visual space.

  • PHOC Auxiliary Head: A PHOC auxiliary prediction head acts as a mild regularizer, encouraging descriptors that share character content to lie near one another.

  • Lexical Space: The word embedding space is built by manifold learning over a candidate vocabulary. Pairwise Levenshtein distances are computed among lexicon entries, and Multi-Dimensional Scaling (MDS) is applied to embed them into a low-dimensional Euclidean space that preserves lexical proximity.

The training process is iterative, moving from a small set of seed word–image pairs to an expanded dataset:

Phase A - Backbone Adaptation: The network fine-tunes the backbone using a mix of synthetic and real aligned samples. The optimization uses the composite loss L HTR = L CTC + lambda L PHOC. This phase is designed to sculpt an initial visual manifold in which distances already reflect lexical relationships.

Phase B - Projector Training: The visual backbone is frozen, and a lightweight projector g is trained. The training objective combines two losses:

  1. L sup: Penalizing the deviation between the projected descriptor i and the embedding of its transcription e y i.

  2. L OT: An entropically-regularized Optimal Transport (OT) objective that globally aligns the empirical measure mu k (of projected image descriptors) with the lexical measure nu (of word embeddings). The projector g is trained with L proj = L sup + lambda OT L OT.

Phase C - Pseudo-Label Expansion: This phase converts soft alignments from Phase B into new training labels. The OT solver provides a distribution over all candidate words for each unlabeled image. Confidence is quantified by measuring the Shannon entropy of its distribution; lower entropy means higher confidence. A fixed budget of top K candidates is selected, and the word with the highest confidence score receives a pseudo-label. These newly labeled items are moved into the aligned set A k, and the training proceeds to the next round.

The algorithm was evaluated on three datasets: GW, IAM, and CVL.

  • Ablation Study (PHOC): Table 1 shows that using CTC + PHOC significantly improves performance over CTC only, especially in low-label scenarios (e.g., at a 1% labeled fraction).

  • Ablation Study (Lexical Prior): Figure 3 demonstrates that the empirical unigram prior is superior to the uniform prior, as the empirical prior concentrates transport where the data lie, yielding more accurate and stable pseudo-labels.

  • Comparison: Table 2 shows that Ours achieves the lowest CER/WER on GW and CVL, noting the largest gains in the low-label regimes.

Improvements for AI systems

The primary failure mode of current state-of-the-art Handwritten Text Recognition (HTR) systems in low-resource settings is their reliance on a massive, manually labeled ground truth (GT). The proposed framework fundamentally improves the learning paradigm by decoupling the need for extensive manual supervision from the ability to achieve high recognition accuracy.

The core improvements are:

  1. Shift from Fully Supervised Learning to OT-Guided Self-Training (Weak Supervision):

  • Mechanism: The system does not wait for human labels. Instead, it iteratively uses Optimal Transport (OT) to align the distribution of visual features (mu k) extracted from unlabeled input images with a fixed distribution of semantic word embeddings (nu).

  • Impact: This alignment generates high-confidence pseudo-labels. The system then retrains on this expanding, self-generated dataset (A k), enabling robust learning even when the initial labeled seed set is minimal.

  1. Integration and Exploitation of Lexical Prior Knowledge:

  • Mechanism: Unlike standard HTR models, a strong prior based on word frequency (e.g., Zipfian skew) is built into the semantic space (nu). The OT objective (Eq. 4) is constrained by this prior (p w), ensuring that the transport plan focuses on matching visual features to words that are statistically likely to occur together.

  • Impact: This provides a structural scaffolding for the training process, dramatically increasing the probability of generating accurate pseudo-labels compared to a uniform or arbitrary alignment approach.

  1. Reframing Recognition as Cross-Modal Alignment (Feature Space Geometry):

  • Mechanism: The system defines two distinct, but related, spaces: a visual space (image descriptors) and a lexical embedding space (word embeddings). By optimizing the OT cost function (L OT), the system is forcing the geometric proximity of these two spaces. This is further regularized by an Auxiliary Prediction Head (PHOC), which ensures that features sharing similar character content are positioned close together in the visual manifold.

  • Impact: The recognition task becomes a matter of finding the closest point in a semantic space to its visual counterpart, rather than simply classifying based on local feature vectors.

The resultant improved HTR system exhibits the following specific capabilities:

  1. High Performance in Low-Resource Regimes: The system can achieve state-of-the-art accuracy (e.g., achieving significantly lower Character Error Rate (CER) and Word Error Rate (WER) than previous self-training methods) using extremely small initial labeled datasets (as low as 1% of the training data).

  2. Robust Handling of Scarce Data: It is specifically designed to operate on historical archives, limited modern collections, or any domain where manual transcription is prohibitively expensive or time-consuming.

  3. Lexicon-Guided Stability: The system's performance is stable and predictable because its pseudo-label generation process is guided by the known frequencies of words in the target vocabulary, preventing random error accumulation common in unsupervised methods.

  4. Decoupled Training/Inference Flow: While OT is used exclusively to guide the training (pseudo-label selection), the final transcription during inference relies solely on a robust, lexicon-free CTC decoder, ensuring that the complex OT alignment only optimizes the training process, not limit the deployment capability.

Abstract

Handwritten Text Recognition (HTR) is a task of central importance in the field of document image understanding. State-of-the-art methods for HTR require the use of extensive annotated sets for training, making them impractical for low-resource domains like historical archives or limited-size modern collections. This paper introduces a novel framework that, unlike the standard HTR model paradigm, can leverage mild prior knowledge of lexical characteristics; this is ideal for scenarios where labeled data are scarce. We propose an iterative bootstrapping approach that aligns visual features extracted from unlabeled images with semantic word representations using Optimal Transport (OT). Starting with a minimal set of labeled examples, the framework iteratively matches word images to text labels, generates pseudo-labels for high-confidence alignments, and retrains the recognizer on the growing dataset. Numerical experiments demonstrate that our iterative visual-semantic alignment scheme significantly improves recognition accuracy on low-resource HTR benchmarks.

Sources

Related papers