Loomis Painter: Learning Painting Processes by Reversing Image-to-Video Generation

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Loomis Painter: Learning Painting Processes by Reversing Image-to-Video Generation".

Jane: Here's the gist of the paper. The gist: The authors propose a unified framework for multi-media painting process generation that employs cross-medium style augmentation and a reverse-painting training strategy to ensure consistent texture evolution and human-aligned generation.

Tom: Sketching and painting are creative processes that are hard to master, but existing instructional videos lack interactivity and personalization because they are inherently passive While recent generative models have advanced artistic image synthesis, they struggle to generalize across media and often show temporal or structural inconsistencies, hindering faithful reproduction of human creative workflows.

Jane: The authors propose a unified framework for multi-media painting process generation with a semantics-driven style control mechanism that embeds multiple media into a diffusion model’s conditional space and uses cross-medium style augmentation.

Tom: This enables consistent texture evolution and process transfer across styles.

Title and authors: Tom: So we’re looking at this paper called "Loomis Painter: Learning Painting Processes by Reversing Image-to-Video Generation." It sounds like they are trying to fix a problem where video generation models don't understand the actual steps involved in painting something.

Jane: Right, so the title suggests they are learning how to reconstruct those painting processes by reversing an image-to-video generation model. It’s focused on taking a finished picture and figuring out what came before it, step by step.

Lu: What this paper is really aiming for is that existing generative models often show these temporal discontinuities and structural jumps when trying to replicate human painting workflows across different media.

Meng: So the core idea here is to get consistent texture evolution and human-aligned generation, which sounds like it solves a real problem for production use where you need repeatable results.

Lalam: I see it focusing on moving beyond just generating an image and into modeling the sequence of actions that created that image across different artistic media.

Tom: Exactly, so they’re not just making a pretty picture; they are trying to model the actual process, which is much more complex than just generating a final look.

Jane: And they’re using this framework to handle multi-media painting, meaning you can input one style and get an output in another, like changing an oil painting into a pencil sketch.

Lu: The implication is that by learning the process in reverse, they might be able to generate much more coherent sequences than current models that just jump from start to finish.

Meng: If it works, it means we could move closer to tools where you can tell the AI exactly how a human would paint something.

The paper's summary: Tom: So the paper explains that they are proposing a unified framework for multi-media painting process generation. They use a semantics-driven style control mechanism that puts different art media into the diffusion model’s conditional space.

Jane: That means instead of treating "oil painting" and "pencil sketch" as totally separate things, they embed both media info into the same system so it understands how to transition between them.

Lu: They use a reverse-painting training strategy which is key here because it helps ensure consistent texture evolution during the generation process.

Meng: So, instead of predicting the next stroke forward in time, they learn to reveal the painting in reverse order, which introduces this kind of monotonic detail removal we talked about before.

Lalam: It sounds like they are training a video diffusion model using this reverse approach to align with human intuition for building up a painting from structure to detail.

Tom: And they use an image-to-video setup where the input image is used to train a Diffusion Transformer with a Flow Matching objective, which is how they get the velocity prediction.

Jane: They also have this art media aware painting process where they create an embedding that combines the artistic medium and scene description into one prompt.

Lu: This embedding drives both stylistic and procedural characteristics, allowing them to synthesize medium-appropriate workflows based on that combined semantic prompt.

The paper's improvements: Tom: They specifically mention two main components in their methodology: cross-media conditioning and reverse-painting learning. Cross-media conditioning infuses medium-aware semantics to guide the strokes and textures.

Jane: And then they have this reverse-painting learning strategy which aligns temporal supervision with human intuition for a progressive buildup from structure to detail.

Lu: The improvement lies in how they structure the training, where they apply a style transformation to obtain an image I*, preserving objects but removing the identity of the original medium.

Meng: That sounds like it’s meant to be a way to decouple the object's structure from its medium so you can control those elements separately.

Tom: And they define this cross-media training loss as LI2V, which is applied to make sure that I*, when conditioned with the text embedding, still follows the velocity prediction of their video generation model.

Jane: And they use a novel metric called the Perceptual Distance Profile, or PDP, to compare the generated sequence against the ground truth.

Lu: The PDP score is computed as an L2 distance between two curves after interpolating raw distance profiles onto a common time axis, which they claim shows their method achieves the best performance across all metrics compared to baselines.

Conclusion: Tom: So, to wrap up, this paper demonstrates that a video diffusion model can successfully recreate the painting process of a given image using this unified framework.

Jane: They show it can sketch outlines, deploy hatching techniques in pencil sketches, use layering techniques in oil painting, and show a natural understanding of objects and their depth.

Lu: The authors have shown that this method enables the model to understand things like sketching outlines versus applying specific hatching or color blocking based on the medium.

Meng: But they do have limitations they flag—for instance, one issue is that an occlusion detector in their data collection pipeline cannot detect hand shadows, which leads to dark artifacts in pencil painting generations.

Lalam: And another thing they mention is that the base model struggles with portraits because it hasn't seen them during training.

Tom: So for now, they have a solid foundation showing the model can replicate the process, but future work needs to focus on fixing those specific issues and expanding what’s possible.