DrawVideo: Grounded and Faithful Multi-Shot Video Generation from Storyboard Keyframe Sketches

arXiv:2605.23508 · cs.GR, cs.AI, cs.CV, cs.MM, eess.IV · Submitted 2026-05-22 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "DrawVideo: Grounded and Faithful Multi-Shot Video Generation from Storyboard Keyframe Sketches".

Jane: As a diligent AI researcher, I have meticulously analyzed both provided summaries of the paper "DrawVideo:

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we're talking about DrawVideo today. It sounds like this is a major step in how we can actually get creative control over making long-form video. This paper tackles the problem of generating videos that have to maintain a consistent story and look right across many minutes, which is something current text-to-video models really struggle with because they just take one big prompt.

Jane: It sounds like the core idea here is breaking down that massive video generation task into smaller, manageable pieces. Instead of trying to make the whole long sequence at once, they propose a method where you control each individual shot separately using different inputs. This makes it much more intuitive for directors to guide the final output.

Lu: Exactly, Jane. The authors are proposing this decomposed approach as a way to give explicit control over things like character pose and camera placement in every single frame of a sequence, which is something previous methods just couldn't handle well <ref:2605.23508#pg0>. They move the focus from one monolithic generation process to a system where structure, appearance, and motion are handled by separate inputs for each shot.

Meng: From an engineering standpoint, that decomposition sounds promising because it limits the complexity of what any single model has to handle at once. If you break it down into shots controlled by sketches and prompts, you can design specialized modules for each part rather than one giant system trying to do everything perfectly <ref:2605.23508#pg1>.

Lalam: I see how this structured approach could really improve how we train generative models for narrative consistency. If the model has to satisfy three distinct, controlled inputs for every shot, it forces a level of discipline in its internal representation that should lead to much more coherent visual narratives in the long run.

Tom: Right, so they lay out this framework called "SketchLongVideo" as a way to do this. They’re not just doing one thing; they are creating a dataset specifically for this sketch-guided video generation, which is a huge contribution because having good data is often the biggest hurdle in AI research <ref:2605.23508#pg2>.

Jane: And the way they build that dataset by aligning raw videos into those specific sketch, appearance, and motion triplets seems like a very thorough way to create training material for this new system. It sounds like they've done a lot of groundwork before proposing the final generation framework.

Title and authors: Lu: The researchers show extensive experiments proving that this method does perform well across several metrics, particularly in structural controllability and intra-shot visual stability, which addresses the drift issues seen in other long video attempts <ref:2605.23508#pg2>. They compare it against methods like SketchVideo and VidSketch, showing where their approach actually offers better results.

Meng: I’m interested in that comparison because those baseline models often struggle with things like character deformation or just not maintaining the scene's identity <ref:2605.23508#pg1>. If DrawVideo can achieve better intra-shot stability, it means the output won't look like a completely different character halfway through a long clip, which is something we need for any production work <ref:2605.23508#pg1>.

Lalam: And when you think about the cultural impact, if we can reliably generate sequences that follow a precise storyboard, it opens up huge possibilities for creating consistent visual media, whether it's animated content or complex visual storytelling in other fields. This level of control suggests that future AI won't just be making random clips anymore; it could be a true director assistant.

Tom: That’s the big picture, Lalam. And moving on to what they suggest for improvement, the authors are already thinking about how to make this framework even more flexible and powerful in the future <ref:2605.23508#pg2>. They acknowledge that while their staged decomposition works well, there's always room to refine how those parts interact.

Jane: So what are these suggested improvements they mention? It seems like they aren't just happy with the current setup; they see ways to push the boundaries further in terms of how the inputs are processed and combined <ref:2605.23508#pg2>.

Lu: They suggest several technical enhancements that focus on improving the prompt decomposition stage. For instance, they think future systems should use a more specialized LLM, fine-tuned on cinematic language, to translate those sparse narrative descriptions into the complex dynamic prompts needed for synthesis Improvement one <ref:2605.23508#pg0>.

Meng: I’m looking at the technical side and see that they want to improve how the sketch is colored. They suggest incorporating learned texture priors during the FLUX generation step, perhaps using a lightweight VAE, so that it captures details like fabric sheen better while keeping the structure from the sketch intact Improvement two <ref:2605.23508#pg0>.

Lalam: That idea of multi-reference latent conditioning sounds incredibly useful for maintaining visual identity across different scenes or characters within a long video. If we can feed in concept art and mood boards simultaneously, it gives the system richer information to anchor itself to Improvement three <ref:2605.23508#pg2>.

Title and authors: Tom: And they are also looking at making the motion decomposition more adaptive rather than fixed. They propose adjusting the number of derivative keyframes based on how complex the initial motion prompt is, which would optimize things computationally without sacrificing precision Improvement four.

Jane: It seems like they are focusing a lot on making the control signals—the sketches and prompts—more robust and more expressive for this multi-shot generation process. This is about tightening up the link between what a director draws and what the AI actually renders Improvement five.

Tom: So, to wrap up this discussion on DrawVideo: it’s a sophisticated method that achieves better control over long video by splitting the problem into controllable shots guided by sketches, appearance prompts, and motion prompts <ref:2605.23508#pg0>. The authors' work with SketchLongVideo provides a valuable dataset to make this possible, and their proposed improvements focus on making the prompting smarter and more adaptable for future applications.

Jane: It really shows how researchers are moving toward systems that can handle complex narrative structures by focusing on controlled decomposition rather than just massive, end-to-end generation <ref:2605.23508#pg1>. It’s a very practical way to tackle the difficulty of long-form video coherence.

Lu: The implications for creative tools are significant because it moves AI toward being a more direct tool for directors, rather than an opaque black box that just spits out results based on vague text <ref:2605.23508#pg1>. We're seeing research that aims to give artists explicit levers to pull in the generation process, which is where the future of generative media lies.

Meng: From a practical standpoint, having these explicit control signals means we can build workflows where we don't have to re-render an entire scene if one small part needs adjustment; we just regenerate that specific shot <ref:2605.23508#pg1>. That’s a huge win for iterative creation.

Lalam: And for the overall culture of AI development, this work emphasizes the need for structured, multi-modal data alignment to achieve high fidelity in complex tasks like long video generation Improvement one <ref:2605.23508#pg0>. It shows that better quality is coming when models are forced to adhere to multiple, specific constraints simultaneously.

Tom: So, we've covered the core idea of DrawVideo and why it’s important for giving directors more explicit control over their visions for long videos. We’re going to take a quick pause before we talk about some of those other interesting papers on arXiv that are shaping the next wave of AI development.

Jane: That sounds like a good time to shift gears and see what else the community is working on right now, Tom. We'll be right back after this short break.

The paper's summary: Tom: So, we've been digging into DrawVideo, and now we're looking at what the researchers actually found in their summary to really grasp how this works in practice for long videos.

Jane: Right, so if I’m understanding correctly, the main thing DrawVideo does is take a big video project and break it down into many smaller shots that each have its own specific set of instructions—a sketch, a look at the character's appearance, and a description of the action happening in that moment.

Lu: Exactly. It moves away from trying to generate the entire movie in one go and instead focuses on making sure every little piece is structurally sound according to the director's initial sketch before adding complex movement details.

Meng: From my side, I’m focusing on how this decomposition limits the computational load per step; it’s much more efficient than trying to manage all those temporal variables at once, which makes sense for real-time or near-real-time generation tasks.

Lalam: And what really stands out is their method of anchoring the appearance using a specialized pipeline that ensures that the visual style and character look stay exactly the same across every single shot, which addresses a major problem we see in current long video attempts.

Tom: That consistency aspect is huge, Jane; if you're making a sixty-minute story, you don't want the main character to suddenly change their clothes or look different between scene one and scene fifty.

Jane: Precisely, Tom. They call this "Sketch Coloring," where they turn that initial sketch into a colored keyframe using an image generation technique guided by ControlNet, which locks in that visual identity early on.

Lu: That anchoring process is smart because it separates the structural information from the stylistic information, allowing them to control the pose and composition separately from how the character looks or what colors are used.

Meng: It sounds like a very robust way to ensure that when you move on to generating those dynamic motion prompts later, you’re starting with a stable visual base instead of something drifting randomly.

Lalam: The implication here is that we can finally get AI tools that can handle the complexity of narrative structure, allowing creators to focus more on the story and less on worrying about technical inconsistencies in long-form content.

Tom: And they've even introduced a whole new dataset called SketchLongVideo, which they say is crucial because having high-quality training data that actually maps these sketch, appearance, and motion triplets is hard to find.

Jane: That dataset construction process sounds very meticulous; by aligning raw videos into those specific inputs, they’re essentially creating perfectly labeled examples for the AI to learn from.

Lu: It really shows how necessary it is to have these structured datasets when you’re aiming for this level of directorial control; it moves us closer to systems that understand narrative structure rather than just pixels.

Meng: I wonder how scalable this is, though; if we want to generate a truly massive feature film, managing millions of these individual shot controls efficiently will be the next big engineering challenge.

Tom: That’s a fair point, Meng. But the results they show across all those evaluation metrics—shot control and consistency—suggest that for now, this staged decomposition is the way forward for achieving that high fidelity we're aiming for.

Jane: It really confirms that by handling the problem in stages, they manage to solve issues like temporal coherence and structural fidelity much better than previous methods.

Lu: I think what’s exciting is how this opens up new creative avenues; imagine a director sketching a complex sequence, and the AI instantly generating a visually faithful interpretation with consistent character identity across every frame.

Tom: It sounds like we are moving toward an era where AI isn't just making pretty clips, but becoming something that functions as a serious collaborative tool for filmmakers and storytellers.

The paper's improvements: Tom: So we’ve seen how DrawVideo tackles the core problem of long video generation by breaking it down into controlled shots, and now we're looking at how they plan to take this framework even further in their suggested improvements.

Jane: It sounds like the authors aren't just satisfied with the current decomposition; they see specific ways to make the inputs themselves more powerful, which is really interesting for future development.

Lu: They are focusing on making the way narrative descriptions get translated into those complex motion prompts much smarter by suggesting a fine-tuned LLM trained specifically on cinematic language instead of relying solely on existing vision-language models.

Meng: That’s practical; if the prompt decomposition is better, the synthesis stage should receive more precise instructions, which means less guesswork for our engineering pipeline when we try to build these tools.

Lalam: And they are pushing beyond just one reference image by suggesting a multi-reference latent conditioning mechanism so that the system can simultaneously manage character identity, scene style, and background mood in every single shot.

Tom: That’s a big deal for consistency; it means we could feed in concept art and a mood board at the same time to guide the generation of one segment.

Jane: I agree, Tom; that level of simultaneous guidance should really help prevent that identity drift we talked about earlier when generating longer sequences.

Lu: They also propose making the motion decomposition adaptive, meaning instead of having a fixed number of action states, the system could dynamically adjust how many keyframes it generates based on how complex the original motion prompt is.

Meng: That adaptation sounds like it would be smart from a resource management standpoint; it lets us optimize computational cost without losing the necessary semantic precision for high-quality motion.

Tom: So they’re essentially saying we should stop using fixed templates and start having the system intelligently decide how detailed to get on a shot-by-shot basis.

Jane: That speaks to a future where the AI isn't just following a rigid blueprint but is actually interpreting the director's intent dynamically at each step.

Lalam: This level of adaptive control suggests that we are moving toward systems that can handle highly nuanced, unpredictable creative direction much more effectively than current models allow.

Lu: These enhancements show a clear path for improving the fidelity of visual storytelling, making the AI capable of handling much more intricate and demanding production briefs down the line.

Tom: It’s exciting to see them thinking about how to make these control signals—the sketches and prompts—more robust across different modalities, which is where we need to focus our research next.

Conclusion: Tom: So we’ve covered the mechanics of DrawVideo, and now we need to wrap up by talking about what this whole endeavor means for creative AI and where we go from here in research.

Jane: Exactly, Tom; essentially, DrawVideo gives us a much more structured way to approach long video generation by forcing it into controllable storyboard shots.

Lu: The implication for the field is that we are seeing a clear direction toward generative systems that can handle high-level directorial intent through explicit structural guidance rather than just vague text prompts.

Meng: From an engineering standpoint, this means we can start building workflows where iteration is easier because you only need to regenerate one specific shot if something in the sequence needs tweaking.

Lalam: For me, the biggest vision here is that we’re moving toward a culture where storytelling isn't just about writing a prompt; it becomes an act of precise architectural design, and this AI framework makes that possible for complex narratives.

Tom: That sounds like something every creative professional in the world needs to hear, Lalam. It gives us tools to actually execute our detailed visions.

Jane: And the evaluation results they show across shot control and consistency really back up the idea that this staged approach delivers on structural faithfulness, which is vital for any professional application.

Lu: The work with SketchLongVideo is a significant contribution because it provides a specialized data set that moves us closer to training models on how to understand narrative structure in video formats.

Meng: I do wonder about the practical deployment scale, though; while the decomposition works beautifully on smaller clips, managing the complexity of generating an entire feature film still presents some massive computational hurdles.

Tom: It’s a valid concern, Meng; but the authors themselves noted that this staged design is specifically aimed at making long-range coherence manageable rather than trying to solve it all in one massive step.

Jane: And their proposed future work on adaptive decomposition shows they aren't stopping there, they’re already thinking about making the system smarter and more efficient for different types of scenes.

Lu: I think the idea of integrating multi-reference latent conditioning is particularly interesting because it opens up pathways for richer, more complex scene management within a single generation pass.

Lalam: If we can achieve this level of visual consistency across long durations, it means AI becomes a more reliable partner in visual media creation, allowing artists to focus on the narrative depth instead of worrying about technical glitches.

Tom: To wrap up, DrawVideo is a very solid framework for achieving director-oriented control through decomposition and appears to set a strong foundation for how we build next-generation long video tools.

Jane: We’ve seen how this paper tackles the challenges of temporal coherence and structural fidelity in long-form video creation using a multi-shot strategy.

Lu: It truly shows that breaking down complex problems into smaller, controllable components is a powerful way to approach generative modeling in media.

Meng: The practical impact lies in enabling more iterative and controllable creative workflows for production teams.

Lalam: Ultimately, this research pushes AI toward becoming a tool for precise narrative construction rather than just an automated clip generator.

Tom: That’s all the time we have for today on DrawVideo; it’s been fantastic diving into this paper with you all.

Jane: It has been a real treat discussing how these technical concepts translate into real creative potential.

Lu: I'm really looking forward to seeing how this framework inspires new architectures in other areas of generative research.

Meng: Keep an eye on those proposed enhancements, because that’s where the next wave of practical application is going to be built.

Lalam: Until next time, remember that precise control over visuals is the key to unlocking deeper creative expression with AI.

The University of Sydney · Charles Sturt University

cs.GR, cs.AI, cs.CV, cs.MM, eess.IV

Submitted: 2026-05-22

Updated: 2026-10-02

Comments: NeurIPS 2026 Workshop on Grounded and Faithful Vision-Language Models for Real-World Deployment

Code: https://github.com/LouckXu/DrawVideo

License: http://creativecommons.org/licenses/by-nc-sa/4.0/

Importance score: 92/100

The gist: As a diligent AI researcher, I have meticulously analyzed both provided summaries of the paper "DrawVideo: Grounded and Faithful Multi-Shot Video Generation from Storyboard Keyframe Sketches." The

Key concepts

Storyboard Decomposition: Breaking down a long video task into many small, manageable shots based on a storyboard. Each shot is controlled by three specific inputs: a sketch for structure, an appearance prompt for look, and a motion prompt for action. This staged approach prevents the model from losing control over complex temporal sequences.
Sketch Coloring (Appearance Anchoring): The initial step where the black-and-white storyboard sketch is converted into a colored reference keyframe. This uses FLUX-based image generation guided by ControlNet to ensure that the structural drawing remains intact while simultaneously assigning a consistent visual style and color palette to the scene, preventing appearance drift later on.
Derivative Keyframes Generation: Decomposing a complex motion description into several intermediate frames, each representing a distinct action or pose change. These frames are generated using conversion prompts that strictly maintain the appearance from the anchor frame while focusing only on specific temporal changes. This allows for fine-grained control over how characters move between major story beats.

Terminology

Consistency — 83× in this paper · explained in 3 episodes
Alignment — 35× in this paper · explained in 1 episode
Fidelity — 14× in this paper · explained in 2 episodes
Diffusion Model — 11× in this paper · explained in 8 episodes
Diffusion Models — 8× in this paper · explained in 6 episodes
Temporal consistency — 8× in this paper · explained in 2 episodes
Conference on Computer Vision and Pattern Recognition — 6× in this paper
Latent Representation — 6× in this paper · explained in 2 episodes
Latent Space — 5× in this paper · explained in 7 episodes
Ablation — 4× in this paper · explained in 1 episode
Core — 4× in this paper · explained in 2 episodes
Semantic Alignment — 4× in this paper · explained in 3 episodes
Ablation study — 3× in this paper · explained in 1 episode
Failure Modes — 3× in this paper · explained in 3 episodes
Grounding — 3× in this paper · explained in 3 episodes
Structural Control — 3× in this paper · explained in 2 episodes
Transformer — 3× in this paper · explained in 2 episodes
Adaptation — 2× in this paper · explained in 1 episode
Decode — 2× in this paper · explained in 2 episodes
Generative Model — 2× in this paper · explained in 2 episodes
Generative Models — 2× in this paper · explained in 2 episodes
Hallucination — 2× in this paper · explained in 3 episodes
Latent Diffusion Model — 2× in this paper · explained in 2 episodes
Latent Diffusion Models — 2× in this paper · explained in 2 episodes
Semantic Similarity — 2× in this paper · explained in 2 episodes
Attention Mechanism — 1× in this paper · explained in 2 episodes
Benchmark — 1× in this paper · explained in 2 episodes
Diffusion Process — 1× in this paper · explained in 2 episodes
Embeddings — 1× in this paper · explained in 2 episodes
Entanglement — 1× in this paper · explained in 3 episodes
F1 Score — 1× in this paper · explained in 3 episodes
Flow Matching — 1× in this paper · explained in 7 episodes
Frustration — 1× in this paper · explained in 2 episodes
Generative Modeling — 1× in this paper · explained in 3 episodes
Multi-agent Collaboration — 1× in this paper · explained in 2 episodes
Prompt Engineering — 1× in this paper · explained in 4 episodes
Provenance — 1× in this paper · explained in 3 episodes
Robustness — 1× in this paper · explained in 5 episodes
Structural Integrity — 1× in this paper · explained in 3 episodes

Summary

As a diligent AI researcher, I have meticulously analyzed both provided summaries of the paper DrawVideo: Grounded and Faithful Multi-Shot Video Generation from Storyboard Keyframe Sketches. The goal is to synthesize these details into a comprehensive, high-fidelity description suitable for a rigorous technical review, ensuring no critical methodological nuance is lost.

Here is the detailed, combined summary:

The paper introduces DrawVideo, a novel long-video generation framework specifically engineered for director-oriented control and storyboard-driven creation. Recognizing the limitations of existing text-to-video methods—which typically struggle with controlling character pose, camera composition, spatial layout, and local motion semantics over extended temporal ranges from a single prompt—DrawVideo fundamentally shifts the paradigm from end-to-end generation to a decomposed, multi-shot storyboard strategy.

DrawVideo decomposes the complex task of long video synthesis into independently controllable storyboard shots. Each shot is governed by three distinct, explicitly controlled inputs:

A Black-and-White Sketch: This serves as the primary structural constraint, dictating pose, composition, and spatial layout for that specific segment.
A Static Appearance Prompt: This defines the visual identity of the scene (character appearance, scene content, and visual style).
A Dynamic Motion Prompt: This guides the shot-level temporal dynamics and action sequence within that shot.

The framework adopts a hierarchical generation strategy termed “global multi-shot, local single-sketch.” This process is meticulously staged to ensure structural fidelity and temporal coherence:

Sketch Coloring (Appearance Anchoring): The initial step involves converting the input storyboard sketch into a colored reference keyframe. This is achieved using a specialized pipeline, leveraging FLUX-based image generation guided by ControlNet, which ensures that the structural integrity of the sketch is preserved while simultaneously producing a stable and stylistically consistent appearance anchor.
Derivative Keyframes Generation (Action State Expansion): The dynamic motion description is then decomposed into multiple derivative keyframes, each representing a discrete action state or transition within the shot. These intermediate frames are generated using conversion prompts designed to describe specific pose changes while strictly preserving the appearance attributes established in the initial anchor frame.
Video Generation (Local Temporal Synthesis): Finally, local video clips are synthesized between adjacent derivative keyframes. This synthesis utilizes a first-last-frame latent video diffusion pipeline, conditioned on the starting and ending derivative keyframes and a structured dynamic prompt, to generate temporally continuous motion for that specific segment.

This staged design is crucial; it explicitly decouples the generation process: the sketch controls structure, the coloring stage establishes appearance anchoring, action state expansion handles temporal semantics, and local synthesis manages fine-grained transitions.

DrawVideo makes three primary contributions to the field:

Novel Framework: It is proposed as the first text-to-long-video generation framework controlled by sparse sketches for director-oriented, storyboard-driven ultra-long video creation.
New Dataset: The paper introduces SketchLongVideo, the first dataset specifically designed for sketch-guided text-to-long video generation. This dataset is constructed by converting raw animation videos into aligned (sketch, appearance, motion) triplets through rigorous processes including shot detection, keyframe extraction, structured vision-language recognition, prompt decomposition, and sketch conversion.
Superior Performance Validation: The framework demonstrates superior performance across critical evaluation metrics compared to established baselines.

The evaluation protocol is exhaustive, assessing performance across four dimensions:

Shot Control: Measured by metrics such as LPIPS, CLIP Image Similarity, and Edge-F1.
Shot Consistency: Assessed via Temporal CLIP Consistency and Temporal LPIPS Consistency.
Story Alignment: Evaluated using Static Keyframe Alignment and Story Frames Alignment.
Shot-level / Local Video Quality: Measured by Event Completion Score and Dynamic Controllability Score.

DrawVideo consistently achieves the best quantitative performance across these dimensions, validated further by human evaluation studies, which report the highest mean opinion scores in structural faithfulness, appearance consistency, storyboard controllability, and overall quality. The ablation study confirms that performance gains are directly attributable to this staged decomposition—specifically the sketch-to-appearance anchor conversion and the use of derivative keyframes for local motion synthesis.

DrawVideo is benchmarked against several leading baselines:

SketchVideo (1kf/2kf): Often fails by producing nearly static videos because a single sketch only provides an initial pose, lacking explicit target action states.
VidSketch: Prone to character deformation, mosaic artifacts, and weak semantic alignment.
FlipSketch: Animates a static drawing but lacks the deep temporal control of DrawVideo's motion decomposition.

Improvements for AI systems

As a fastidious and diligent researcher, I have analyzed the DrawVideo framework and its associated data, SketchLongVideo. The core innovation lies in decomposing long-video generation into independently controllable storyboard shots using a structured sketch-guided, appearance-prompted, motion-prompted triplet paradigm.

Based on this scientific foundation, here are specific improvements for AI systems and the capabilities they can unlock:

) Specific Improvements for AI Systems

Decomposition of Control Signals (The Why):
Hierarchical Prompt Structuring (The How):
Deterministic Structural Anchoring (The Constraint):
Reference-Conditioned Temporal Expansion (The Action State Control):
Structured Video Synthesis for Coherence (The Execution):

) Capabilities of the Improved AI System

Director-Oriented, Ultra-Long Narrative Generation: The system can now generate feature films or extended animation sequences guided solely by director sketches and high-level narrative prompts, moving beyond simple text-to-video.
Precise Spatial and Compositional Control: Users can dictate exact camera angles (through the sketch), character poses, spatial layouts, and scene compositions frame-by-frame, ensuring directorial vision is translated into visual reality rather than relying on implicit diffusion models.
High Intra-Shot Visual Stability: By anchoring each shot with a colored keyframe derived from the sketch and appearance prompt (Sketch Coloring module), the system eliminates identity drift and style variation within a single shot, ensuring characters look consistent across all generated frames of that specific scene.
Fine-Grained Action Semantics Control: The ability to decompose motion prompts into discrete action states (Initialization, Forward Movement, Peak Motion) allows directors to choreograph complex sequences with precise timing and intent, leading to highly controllable local dynamics rather than generic motion interpolation.
Robust Long-Range Coherence: By synthesizing short video clips between keyframes using a first-last-frame conditioning module (Wan 2.2), the system maintains temporal continuity across the entire long narrative, preventing drift over extended durations—a critical failure point for current long video models.

) Specific Technical Enhancements

Enhancing Prompt Decomposition (Section B): The current decomposition relies on VLM recognition (LLaVA-OneVision). Future systems should integrate a more robust, fine-tuned LLM specifically trained on cinematic language to better translate sparse narrative descriptions into the complex, multi-component structured dynamic prompts required for the synthesis stage.
Advanced Sketch Coloring Fidelity (Section C): While Canny edge maps are efficient, incorporating learned texture priors (perhaps via a lightweight VAE or a specialized style encoder) during the FLUX generation step could allow the generated keyframe to capture subtle material properties (like fabric sheen or surface roughness) more accurately while retaining the sketch's structural integrity.
Multi-Reference Latent Conditioning: Expand beyond single-reference conditioning. Implement a mechanism that allows multiple reference images (e.g., character concept art, background mood boards) to be encoded and concatenated into the ReferenceLatent module, enabling simultaneous control over identity, style, and scene elements in a single shot generation process.
Adaptive Decomposition Depth: Instead of a fixed five-stage decomposition for motion (Section B), implement an adaptive mechanism where the number of derivative keyframes is dynamically adjusted based on the complexity score derived from the initial motion prompt enhancement, optimizing computational cost while maintaining semantic precision.

Abstract

Long video generation requires high-fidelity visual synthesis, coherent narrative organization, shot-level structure, and explicit user control. Existing text-to-video methods typically generate videos from a single long-form prompt, making it difficult for creators to directly control character pose, camera composition, spatial layout, and local motion. We propose DrawVideo, a sketch-guided and storyboard-driven framework that grounds multi-shot video generation in creator-provided spatial structure, appearance, and motion. Each shot is specified by a black-and-white sketch, an appearance prompt, and a motion prompt. DrawVideo first generates a structure-aligned reference keyframe, expands the motion description into derivative keyframes representing ordered action states, and then synthesizes local video clips between adjacent keyframes. We further introduce SketchLongVideo, to the best of our knowledge the first evaluation dataset of ordered multi-shot storyboards with aligned sketch, appearance, and motion conditions. Extensive experiments demonstrate strong structural controllability, appearance consistency, intra-shot visual stability, and motion alignment, providing an effective solution for director-oriented video creation.