From Sketch to Scene: Relightable Garment Reconstruction from Marked Sketches

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "From Sketch to Scene".

Jane: The gist The system transforms a rough garment sketch,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Moving on to the title and authors of this paper, "From Sketch to Scene: Relightable Garment Reconstruction from Marked Sketches". It’s clear they are focusing on taking a very simple input—a sketch with just a star for highlights and a moon for shadows—and turning it into something much more complex than what previous systems could do.

Jane: That title really tells you the main selling point, which is that it’s not just about getting a three dee model; it’s specifically about reconstruction and making sure the lighting is relightable <ref:2610.21865#pg1>. It moves beyond just generating geometry to defining how that geometry interacts with light.

Lu: They are trying to show that sketching can be a more direct way to capture spatial structure and intent, unlike text prompts which current systems rely on heavily >

Meng: So they’re arguing that the sketch captures the designer's specific intent for highlights and shadows directly, whereas text prompts are more abstract >

Tom: Right. The authors are Dogan, Ba¸saran, Sezgin, and Montazeri from Koç University and Manchester University. They cite their work on eliciting that two-symbol notation from art students as a key part of their methodology >

Jane: And they emphasize that they conducted an elicitation study with art students to derive the star for a highlight and the moon for a shadow, which shows this isn't just some arbitrary choice but something derived from real designer input >

Lu: It’s important because it grounds the notation in how actual designers think about visual ideas, moving it away from just being an abstract concept >

Meng: From what I see, they are setting up a solid foundation by defining exactly what those symbols mean and how they should be placed on the drawing itself >

Tom: So, if we look at this as a whole, the authors are building a pipeline that starts with these two simple marks and uses them to control everything from the final light direction to the surface displacement >

Jane: It’s about showing how those two symbols can be used as powerful controls instead of just being decorative elements on a drawing >

Lu: They are moving away from just having the shading baked into a texture, which is what they call "a nuisance" in their related work because you can't move the illumination afterward >

Meng: So this paper is proposing a system that treats those marks as an instruction set for both geometry and lighting simultaneously >

Tom: Exactly. The authors are proposing that instead of treating shading as something to be removed or baked in, they treat it as a direct input for generating the final scene structure >

Jane: So the big implication here is that we have a system where the light isn't fixed in the texture and can be moved and edited after generation >

Lu: It opens up possibilities because you’re not locked into one lighting scenario when you generate it; you’re getting a starting point that's inherently editable >

Meng: So this paper is showing how to build a system where the designer has control over the light in a more fundamental way than just picking a color map >

The paper's summary: Tom: Now let’s look at the actual summary of "From Sketch to Scene: Relightable Garment Reconstruction from Marked Sketches". It boils down to them conducting an elicitation study with art students to get their notation, which is the star for highlight and moon for shadow >

Jane: So they also detail how they predict that scene's directional light from those marks, which is a unit vector, and then they predict a displacement field from the marks to make sure the moon-marked regions fall into shadow under that predicted light >

Lu: And they combine these into a pipeline that turns a marked sketch and a texture prompt into this relightable three dee scene > <ref:2610.21865#pg1>

Meng: They also talk about how they synthesize the albedo separately, making sure the base color doesn't have illumination baked in so you can apply the predicted irradiance later >

Tom: So they’re essentially giving us a complete system that takes a sketch and a texture prompt and outputs geometry, material, lighting, and displacement information >

Jane: It’s a comprehensive pipeline where every part—the lighting prediction, the surface bending—is driven by those initial marks on the sketch >

Lu: They are showing how to use those marks as the input to drive both the light direction and the geometry simultaneously >

Meng: They are moving beyond just getting a single color map and adding this explicit directional light component, which is what makes it different from other texture-only stages >

Tom: So this paper’s main summary is that they create a system where you get an output that includes an explicit directional source ready to be relit >

Jane: It’s a complete scene with geometry, material properties, and the predicted lighting direction all bundled together for easy editing later >

The paper's improvements: Tom: Let’s look at the specific improvements they suggest in "From Sketch to Scene: Relightable Garment Reconstruction from Marked Sketches". They point out that a major improvement is treating shading marks as a first-class input, which we discussed >

Jane: And they also highlight how predicting the directional light from those marks allows for an editable environment, meaning the light is no longer fixed in the texture >

Lu: Another key thing they show is predicting a displacement field separately to handle shadows explicitly so that those moon-marked regions fall into shadow under that predicted light instead of being painted dark in the texture >

Meng: And they also mention how they combine these into a pipeline that turns the marked sketch and texture prompt into this fully specified glTF scene >

Tom: They are showing how to combine these components into one cohesive pipeline that handles the albedo synthesis, lighting prediction, and displacement field generation all at once >

Jane: And they also highlight how they separate the albedo synthesis so you can apply the predicted irradiance afterwards instead of baking it into the color >

Lu: So they are showing how to disentangling those parts is crucial because you can’t have illumination left in the texture if you want to move it later >

Meng: It really shows that separating them up makes sense for production because it prevents getting stuck with baked-in lighting issues >

Tom: The results from DeepFashionthree dee show that the light direction was recovered to twenty-two point nine degrees mean and forty-seven point five degrees median angular error, which is pretty concrete evidence of the system working on real data >

Jane: And they also showed that darkening at the marks showed that seventy-nine percent of the moon marks are darker than before, against forty-four percent of equally sized discs placed away from any mark on the same garments >

Conclusion: Tom: So to wrap up, this paper "From Sketch to Scene: Relightable Garment Reconstruction from Marked Sketches" is about using a marked sketch input to get a relightable garment scene with an explicit directional light and geometry-driven shadows that you can edit later >

Jane: The big takeaway is that designers can specify lighting intent directly on the sketch, which changes how we think about generating three dee assets from sketches > <ref:2610.21865#pg1>

Lu: It validates the idea that these simple symbols are powerful enough to drive complex features like light and surface geometry simultaneously without needing a lot of extra training for every single mark >

Meng: For deployment, it means we can build systems that respect artistic intent regarding light and shadow in a more fundamental way than just relying on texture maps >

Lalam: I just want to say that the paper shows how this pipeline works by turning a marked sketch and a texture prompt into an output with the predicted light source as an explicit directional source alongside the textured mesh and the displacement field for accurate shadowing >

Tom: So it’s a system that takes those simple marks and gives you something fully specified, which is definitely worth looking at for anyone working in three dee generation or asset creation > <ref:2610.21865#pg1>

Jane: We’ve covered how this new approach lets designers specify lighting intent directly on the sketch rather than relying on baked-in textures >

Lu: We’ve seen how the paper validates that these simple symbols are capable of driving complex features like light and surface geometry without needing a lot of extra training for every single mark >

Meng: And for production, it means we can build systems that respect artistic intent regarding light and shadow in a more fundamental way than just relying on texture maps >

Lalam: So the "From Sketch to Scene: Relightable Garment Reconstruction from Marked Sketches" paper is about getting an output with the predicted light source as an explicit directional source alongside the textured mesh and the displacement field for accurate shadowing >