InkSight: Offline-to-Online Handwriting Conversion by Teaching Vision-Language Models to Read and Write

summary

The gist

The gist: The work proposes InkSight, which empowers physical note-takers to effortlessly convert their offline handwriting into digital ink by leveraging vision-language models to combine reading

In short

InkSight converts offline handwriting from photos into digital ink using vision-language models that combine reading and writing knowledge. It overcomes the lack of paired data by training a multi-task model on recognition and derendering tasks. The system effectively transforms arbitrary handwritten text in diverse images into traceable digital strokes, showing high accuracy in human evaluation.

Key concepts

Derendering: This is the core process of InkSight: transforming an input photo of handwriting (offline) into a sequence of digital ink strokes (online). It bridges the gap between physical notes and digital formats by turning visual handwriting into a structured, editable digital representation.
Reading and Writing Priors: The model leverages learned knowledge from both reading text and writing strokes. Instead of relying only on geometric image properties, this approach uses these combined priors to train the model, allowing it to generalize better even when paired image-ink samples are scarce.
Digital Ink Representation: Digital ink is represented as a sequence of strokes, where each stroke is defined by a series of coordinate-time triplets. To make this usable by sequence models like mT5, the ink data is first normalized through resampling and position adjustments onto a fixed canvas.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 19 episodes
Federated Learning — explained in 17 episodes
Retrieval-Augmented Generation (RAG) — explained in 17 episodes
Quantum Fisher Information (QFI) — explained in 16 episodes
Interpretability — explained in 15 episodes

This episode discusses

The paper

InkSight: Offline-to-Online Handwriting Conversion by Teaching Vision-Language Models to Read and Write · Read on arXiv

Blagoj Mitrevski, Arina Rak, Julian Schnitzler, Chengkun Li, Andrii Maksai

Google DeepMind · EPFL

Digital note-taking is gaining popularity, offering a durable, editable, and easily indexable way of storing notes in a vectorized form, known as digital ink. However, a substantial gap remains between this way of note-taking and traditional pen-and-paper note-taking, a practice that is still favored by a vast majority. Our work InkSight, aims to bridge the gap by empowering physical note-takers to effortlessly convert their work (offline handwriting) to digital ink (online handwriting), a process we refer to as derendering. Prior research on the topic has focused on the geometric properties of images, resulting in limited generalization beyond their training domains. Our approach combines reading and writing priors, allowing training a model in the absence of large amounts of paired samples, which are difficult to obtain. To our knowledge, this is the first work that effectively derenders handwritten text in arbitrary photos with diverse visual characteristics and backgrounds. Furthermore, it generalizes beyond its training domain into simple sketches. Our human evaluation reveals that 87% of the samples produced by our model on the challenging HierText dataset are considered as a valid tracing of the input image and 67% look like a pen trajectory traced by a human.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "InkSight: Offline-to-Online Handwriting Conversion by Teaching Vision-Language Models to Read and Write".

Jane: The gist: The work proposes InkSight, which empowers physical note-takers to effortlessly convert their offline handwriting into digital ink by leveraging vision-language models to combine reading and writing priors.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So to wrap up "InkSight: Offline-to-Online Handwriting Conversion by Teaching Vision-Language Models to Read and Write", the authors presented the first approach we’ve seen for converting photos of handwriting into digital ink using standard building blocks.

Jane: They achieved this by teaching vision-language models to read and write together, rather than relying on just geometric properties of the images.

Lu: The key finding is that combining reading and writing priors allows them to train a model effectively even when they don't have large amounts of paired image and ink samples.

Meng: What this means for deployment is that it’s not just about getting an accurate text extraction; it’s about preserving the rich structural information embedded in the handwriting, like stroke order.

Tom: They showed that their models can produce results that accurately reflect the text content while ignoring semantically irrelevant background details.

Jane: The human evaluations confirmed this, with eighty-seven percent of their model's output being considered a valid tracing of the input image.

Lu: But they also found sixty-seven percent looked like a pen trajectory traced by a human, which is how the digital ink representation works.

Tom: The work provides weights and code for the Small-p model, which is useful for other researchers exploring this area or even for building better recognition systems.

Jane: It’s an approach that can serve as a powerful geometric pseudo-label for unsupervised analysis, offering a language-agnostic alternative to analyzing things purely through OCR.

Conclusion: Tom: So, we’re wrapping up on InkSight: Offline-to-Online Handwriting Conversion by Teaching Vision-Language Models to Read and Write. Basically, they figured out a way to take a messy photo of handwriting and turn it into digital ink that actually looks like real writing.

Jane: It’s about bridging that gap between what people actually write on paper versus what computers can process online. The authors use vision-language models to teach the system how to read and how to write at the same time, which is a big shift from just looking at the shapes of the ink.

Lu: They’re using these reading and writing priors to train their model, which means they don't need massive datasets of perfectly paired images and ink samples to get started. That's a key part of their approach.

Meng: From a deployment standpoint, the fact that it scales well on arbitrary input images is important. We’re looking at how much computational overhead this derendering process adds when we try to run it in production environments with high throughput.

Lalam: I see the core idea: transforming an image into a sequence of strokes, where each stroke has coordinates over time, which lets the transformer model generate that ink sequence. It’s essentially a structured way to represent handwritten text digitally.

Tom: The results are pretty strong on the HierText dataset, showing that eighty-seven percent of their output is considered a valid tracing of the original image. That’s solid data for them to work with.

Jane: And they also found that sixty-seven percent of those generated strokes look like something a human actually traced by hand. That’s a pretty impressive metric for how well it captures the writing style without just being pure geometry.

Lu: They also showed it generalizes beyond the training images, meaning it works on different styles and even simple sketches they weren't explicitly trained on. That shows the priors are really learning something about handwriting itself.

Meng: The limitation they mention is that context length in transformer models still limits how much text you can derender in a single go. So, for long documents, you’d probably need to break them up into smaller pieces to process them all at once.

Tom: That makes sense for inference speed. But the authors are releasing the Small-p model weights and code, which is useful if other researchers want to build on this foundation or test it further.

Jane: It opens up a path for using this digital ink sequence as a kind of pseudo-label for analyzing handwriting without needing to run full optical character recognition every single time. It’s not just about the text; it’s about the structure of the writing itself.

Lu: Exactly, it gives you a language-agnostic way to look at handwriting patterns before you even try to read what those patterns mean as words.

Tom: So, InkSight isn't just another OCR tool; it’s a new way to represent physical handwriting digitally that carries structural information.

Jane: It’s definitely something worth watching if you're interested in how we can make digital text more aware of the physical act of writing.

More episodes

← Home