VideoWeaver: Evaluating and Evolving Skills for Agentic Long Video Generation

VideoWeaver introduces an agent harness and benchmark designed to evaluate and evolve skills for long video generation, addressing the gap in understanding how general-purpose agent frameworks can handle complex, long-horizon multimodal tasks by enabling agents to compose their own workflows rather than following predefined pipelines.

The gist

VideoWeaver is an agent harness and benchmark that evaluates and evolves skills for long video generation, where an agent turns a single instruction into a long video by composing foundation skills into its own workflow rather than following a predefined pipeline.

Benchmark Construction and Evaluation Methodology

The benchmark consists of 16 task categories and 285 cases, spanning text, image, audio, video, and their combinations. The dataset is split into in-distribution train, test sets, and three out-of-distribution categories to assess both in-distribution performance and generalization. Evaluating these agents is difficult because errors can occur at any stage of the generation process, necessitating a novel approach. To address this process-level failure diagnosis, the paper proposes an agent-as-judge that inspects both the execution trace and the final video, grounding its scores in evidence such as metadata and intermediate files.

Skill Evolution Algorithm

The core innovation lies in refining agent capabilities through feedback. The proposed skill evolution algorithm is designed to progressively refine and merge skills. This process involves three stages:

1. Task Inference: The execution agent generates an initial category-level composition skill, denoted as S0,k.

2. Skill Evolution: The optimizer agent iteratively refines the composition skill and creator skill using feedback from the evaluation agent to obtain optimized skills, such as S⋆ = arg max Sk X x∈Dtest k ϕ x, Aexec(x, Sk).

3. Creator Skill Merging: The merge agent integrates experience across different categories into a single creator skill by progressively disclosing and merging category-level creator skills.

Metrics and Results

The evaluation utilizes two main sets of metrics: process metrics and output metrics. Process metrics check execution checkpoints in the trace, such as Planning PL and Successful tool calls / total tool calls EE. Output metrics assess quality, emphasizing cross-clip consistency, which includes Visual Consistency VC (characters and scenes remain consistent across clips) and Audio Consistency AC (speaker identity and background music style remain consistent across clips). Experiments show that an explicit composition skill improves the generation process over using foundation skills alone, while the skill evolution further improves output quality. Furthermore, a human study confirms that the proposed agent-as-judge aligns well with human judgments, especially on process metrics.

Foundation Skills and Composition Skills

The harness environment equips a backbone language model with a set of foundation skills, self-contained, independently invocable capabilities for video, image, and audio generation, understanding, and media processing. These are formalized into foundation skills (e.g., generating video clips or synthesizing images) that form the primitive action space. Beyond these isolated capabilities lies the concept of composition skills, which are defined as high-level procedural policies that specify how foundation skills are orchestrated to accomplish such long-horizon generation tasks. The system also defines a skill-creator as a skill that constructs composition skills from available foundation skills and task cases.

Comparative Analysis

Experiments across multiple agent frameworks and foundation models reveal performance variations. Performance varies notably across harness and model choices, with mature industrial settings achieving the strongest overall results on both process and output metrics. The comparison between No Composition Skill and Composition Skill demonstrates that foundation skills alone are insufficient, as the latter shows improvements in critical steps like Input Processing IP and Planning PL. The final results indicate that Evolution with Feedback achieves a higher output average than Self-Evolution, suggesting that external feedback provides valuable output metrics feedbacks harder for the agent to self-evaluate from its own execution.

Human Alignment and Limitations

The agent-as-judge is validated against human annotations, showing alignment scores are higher for process metrics and FR mainly because they are easier to evaluate than open-ended output quality. Limitations include the current dataset size due to computational costs, the foundation skills being primarily from the ByteDance ecosystem, and future work focusing on more principled optimization methods and stronger skill verification. Potential risks involve intellectual property concerns regarding copyrighted assets and substantial computational cost.

Foundation Composition Skills

The set of foundation skills includes capabilities like:

(List extracted from Table 9)

(Examples include: AI Avatar Video Generate, One-Sentence Anime Video, Object Evolution Video, Cinematic Video, Audio-Driven Story, Long Video Edit.)

Foundation Composition Skills (Detailed)

The provided foundation skills expose basic video, image, audio, understanding, metadata, and mediaprocessing capabilities. For generation skills (e.g.

As a fastidious and diligent researcher, I have analyzed the VideoWeaver: Evaluating and Evolving Skills for Agentic Long Video Generation paper. The core innovation lies in moving from fixed pipelines to agent-composed workflows, supported by an evidence-grounded evaluation loop (agent-as-judge) and a skill evolution algorithm.

Based on this research, here are specific improvements that can be made to AI systems, categorized by the capability they enable:

The improved AI system will operate as a highly autonomous, long-horizon creative director capable of generating complex cinematic narratives and personalized media from minimal high-level instructions. It moves beyond simple tool use to perform complex procedural planning and self-correction across multiple generative steps.

Here are the specific improvements:

1. A system that can turn a single high-level instruction (e.g., Create a 5-minute cinematic vlog about my trip to Tokyo) into a complete, multi-shot video by dynamically composing necessary foundation skills into its own custom workflow, rather than following a pre-set sequence.

2. The ability to perform long-horizon planning by decomposing the instruction into intermediate objectives, selecting and sequencing the appropriate foundation skills (video generation, image generation, audio synthesis), managing intermediate artifacts across these steps (e.g., character consistency data), and ensuring narrative, visual, and audio consistency across all resulting clips.

3. A robust evaluation mechanism that doesn't just check the final output quality but actively inspects the entire execution trace—including planning steps and tool calls—to diagnose failures at any intermediate stage (e.g., a flawed plan or a failed tool call).

4. An Agent-as-Judge that uses extracted evidence from the execution trace (metadata, intermediate files) to ground its scoring on process metrics (like Execution Error-Free Rate and Clip Merging) and output metrics (like Visual Consistency, which specifically checks for character consistency across clips).

5. A self-refining skill acquisition loop where the agent uses the feedback from the Agent-as-Judge to progressively refine its composition skills (how it sequences foundation skills) and creator skills (how it structures the overall generation process), leading to higher quality outputs and better generalization to unseen tasks.

6. The ability to adapt its orchestration strategy based on learned experience: explicit composition skills are shown to improve generation over using foundation skills alone, indicating the system learns when a specific procedural skill is superior for a given task type.

7. Superior performance in complex, multimodal tasks where coherence across different modalities (e.g., ensuring the audio track matches the visual scene transitions) is critical.

This improved AI system can specifically:

• Generate long-form, coherent videos (minutes to hours long) that adhere to complex narrative constraints.

• Handle intricate cross-modal dependencies (e.g., matching a specific character's appearance across multiple shots while ensuring the background music style remains consistent).

• Self-correct its generation process in real-time by inspecting execution logs and intermediate video frames, fixing planning errors or tool call failures before the final output is compromised.

• Generalize to entirely new, unseen long video generation tasks by transferring learned orchestration knowledge from training data to out-of-distribution scenarios.

Agentic long video generation requires planning, tool orchestration, and cross-clip coordination over a long horizon. Most existing video agents either rely on static, human-crafted workflows, which require substantial manual effort and poorly adapt across tasks, or iteratively refine the output of the current task without persistently distilling execution experience into reusable skills for future tasks. We introduce VideoWeaver, an agent harness and benchmark that evaluates and evolves skills for long video generation. Given a single high-level instruction, an agent dynamically composes foundation skills into its own workflow rather than following a predefined pipeline. We construct a benchmark of 16 task categories and 285 cases, with references spanning text, image, audio, video, and their combinations. We further propose an evidence-grounded agent-as-judge that inspects both the execution trace and the final video to diagnose process and output failures. Based on this feedback, our evolution algorithm progressively refines category-level composition and creator skills, allowing recurring experience to guide dynamically constructed workflows for unseen cases. Experiments show that explicit composition skills improve the generation process over foundation skills alone, while skill evolution further improves output quality and generalizes to unseen cases. Incorporating judge feedback yields additional gains, especially on output metrics, and the agent-as-judge aligns well with human, particularly on process metrics. Code is available at https://github.com/JianhuiWei7/VideoWeaver.