VISD: Enhancing Video Reasoning via Structured Self-Distillation

arXiv:2605.06094 · cs.CV, cs.AI · Submitted 2026-08-20 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "VISD: Enhancing Video Reasoning via Structured Self-Distillation".

Tom: Based on the provided text, here is a long and detailed summary of the scientific paper: VISD:

Jane: First, who's behind it and why it matters.

Title and authors: Tom: Well, moving onto the specifics of the paper "VISD: Enhancing Video Reasoning via Structured Self-Distillation," they introduce a video-aware judge model, which is their main tool for generating this structured feedback. This judge model evaluates each reasoning trajectory and spits out feedback on several dimensions.

Jane: That sounds like they are moving away from simple pass or fail signals and creating a richer set of diagnostic information. They’re looking at answer correctness, logical consistency, and how well the reasoning is grounded in the video data itself.

Lu: Precisely! By decomposing the quality into those different dimensions—correctness, logic, and spatio-temporal grounding—they are giving the model much more detailed insight into what went wrong.

Meng: So they are essentially building a specialized critic for video reasoning that doesn't just check the final output but scrutinizes the entire reasoning path for specific types of errors like temporal misalignment.

Lalam: It’s about creating a privileged context, r, which bundles this detailed feedback along with verified answer information and available grounding evidence, allowing the teacher model to rescore things in a much more informed way.

The paper's summary: Tom: And that structured feedback is then put to work by using a direction–magnitude decoupling mechanism to integrate it with the reinforcement learning rewards they’ve already been using. This is where they try to solve the instability issues we talked about earlier.

Jane: I remember hearing something about separating the update direction from the magnitude modulation, which is smart because it keeps the core RL signal—the advantage from the environment reward—controlling *where* to go in terms of policy change.

Lu: That separation allows them to use the structured information not just as a general nudge, but specifically to modulate how strongly each token gets updated based on its specific diagnostic quality.

Meng: So if the environment reward says we should generally move in one direction, the structured signal tells us how much weight to put on specific tokens based on their quality—that’s a very granular form of credit assignment.

Lalam: That decoupling is key because it lets them have both the broad guidance from RL and the fine-grained corrections from self-distillation without letting one completely overwhelm the other.

The paper's improvements: Tom: The optimization strategy they employ involves a curriculum scheduling approach, gradually shifting the model's reliance from this new structured self-distillation to pure reinforcement learning over time. They also stabilize the teacher model using an exponential moving average.

Jane: That sounds like a very thoughtful way to train long sequences; starting with dense supervision and slowly weaning off that while keeping a stable reference point is a delicate balancing act.

Lu: The curriculum schedule controlled by lambda s manages that transition between the two supervision sources, which helps ensure the model learns the necessary skills sequentially rather than getting overwhelmed by both at once.

Meng: From an engineering perspective, managing that transition smoothly is important because abrupt changes in supervision can cause massive spikes in loss or even policy instability during training on long video sequences.

Lalam: And the EMA teacher stabilizes the signal itself, which I think is important because if the teacher's score keeps fluctuating wildly, it makes it impossible for the student to learn a consistent target.

Conclusion: Tom: Alright team, we’ve covered a lot on how VISD uses structured self-distillation to give us better diagnosis and stability in video reasoning. It seems they’ve managed to keep the dense supervision while keeping the RL framework stable through careful scheduling and decoupling.

Jane: So, in short, this paper proposes a framework that turns auxiliary supervision into meaningful, multi-dimensional data that guides token-level learning more effectively than what was possible before.

Lu: It really opens up possibilities for how we can interpret complex video reasoning behaviors because they are explicitly identifying logical inconsistencies and grounding failures separately.

Meng: For practical impact, this means that when we deploy VideoLLMs, we might be able to diagnose precisely why a model failed on a specific task, which is something we need for building reliable systems.

Lalam: I'm excited because if we can build models that truly understand the structure of visual evidence and reasoning paths this way, it could significantly improve how our AI interacts with and interprets real-world video content in ways that are currently just theoretical concepts.

Tom: Absolutely, it’s a solid piece of work. We’re going to take a quick break before we look at what these advances mean for the broader world and what comes next in this field.

Jane: Stay with us; we'll be right back after the break to talk about the implications of VISD.

Hao Lin, Kunyang Lv, Xu Jiang, Jingqi Tian, Zhongjing Du, Jiayu Ding, Qiaoman Zhang, Hongbo Jin

Hustun University of Science and Technology (HUST) · Wuhan University · Peking University · Tsinghua University

cs.CV, cs.AI

Submitted: 2026-08-20

Updated: 2026-08-21

Project page: https://lkyyy111.github.io/VISD

Importance score: 83/100

The gist: Based on the provided text, here is a long and detailed summary of the scientific paper: Problem Statement and Motivation Training Video Large Language Models (VideoLLMs) for complex reasoning is

Key concepts

Video-aware judge model
This is the main tool in VISD that evaluates reasoning trajectories. It generates structured feedback on multiple dimensions, including answer correctness, logical consistency, and how well the reasoning is grounded in the video data itself.
Direction–magnitude decoupling mechanism
This mechanism separates the update direction from magnitude modulation. This allows the core reinforcement learning signal to control where to move in policy changes, while structured information modulates how strongly each specific token gets updated based on its diagnostic quality.
Curriculum scheduling approach
The optimization strategy gradually shifts the model's reliance from structured self-distillation to pure reinforcement learning over time. This is managed by a schedule controlled by lambda s to ensure sequential skill learning without overwhelming the model with both supervision sources simultaneously.

Terminology

Summary

Based on the provided text, here is a long and detailed summary of the scientific paper:

Problem Statement and Motivation

Training Video Large Language Models (VideoLLMs) for complex reasoning is challenging because it requires align[ing] long reasoning trajectories with temporally evolving visual evidence, where both the final answer's correctness and the faithfulness of intermediate steps are critical. A major difficulty lies in that learning signals must not only indicate if an output is correct, but also identify where and why the reasoning process succeeds or fails.

Existing approaches suffer from limitations:

  1. Reinforcement Learning with Verifiable Rewards (RLVR): While providing reliable supervision, these rewards are inherently sparse and operate at the sequence level, failing to capture the heterogeneous contributions of individual tokens within long reasoning chains.

  2. Self-Distillation: Existing methods often treat auxiliary supervision as unstructured or modality-agnostic signals, lacking diagnostic specificity to distinguish between errors (e.g., logical inconsistency vs. grounding failure) and often interacting unstably with reinforcement learning.

The VISD Framework: Structured Self-Distillation

VISD proposes a structured self-distillation framework that introduces diagnostically meaningful privileged information for video reasoning. The core idea is to elevate self-distillation from a generic auxiliary signal to a structured supervision space.

1. Structured Privileged Information Generation (The Judge Model)

VISD introduces a video-aware judge model (J) that evaluates each sampled reasoning trajectory and produces structured feedback (f). This feedback encapsulates multiple complementary aspects of evaluation, including:

  • Answer correctness.

  • Logical consistency between the reasoning process and the answer.

  • Spatio-temporal grounding quality, which specifically assesses temporal alignment and spatial grounding.

This judge output is packaged with verified answer-side information (a) and available grounding evidence (e) to form a privileged context r = (a, e, f). This allows the teacher model to rescore the same student completion using this detailed diagnostic context.

2. Direction–Magnitude Decoupled Structured Self-Distillation

To integrate dense supervision with RL stability, VISD employs a direction–magnitude decoupling mechanism. The overall training objective follows a policy-gradient formulation:

grad theta J = E y about pi theta (timesx) t grad theta pi theta (y t x, y<t)

  • Direction Determination: The rollout-level advantages (A), computed from the environment reward R(x, y), determine the update direction of for policy updates.

  • Magnitude Modulation: The structured privileged signals (t), derived from the judge's feedback, modulate the magnitude of token-level updates.

This separation allows for semantically aligned and fine-grained credit assignment. The token-level reweighting factor (w t) is defined as:

w t = (sign(A) times t)

To ensure stability, this factor is clipped:

m t = clip(w t, 1 - epsilon w, 1 + epsilon w)

The final token advantage (t) integrates both the environment reward and the structured signal:

t = A(i) (1 - lambda s) + lambda s m t

3. Stable Policy Optimization for Long Video Sequences

To manage long-horizon dependencies and heterogeneous rewards, VISD employs two strategies:

  • Curriculum Scheduling: The training process gradually transitions from structured self-distillation to reinforcement learning, controlled by the mixing coefficient lambda s.

  • EMA Teacher Stabilization: An exponential moving average (EMA) version of the policy parameters is maintained to construct the teacher model:

from tau + (1 - tau) theta

This design reduces high-frequency fluctuations in the teacher signal and mitigates instability caused by rapidly changing policy distributions.

Performance and Conclusion

Experiments on diverse benchmarks show that VISD consistently outperforms strong baselines, improving answer accuracy and spatio-temporal grounding quality. Notably, VISD achieves these gains with nearly 2 times faster convergence in optimization steps, demonstrating the effectiveness of structured self-supervision in improving both performance and sample efficiency for VideoLLMs.

(Note: The summary is constructed by synthesizing information from the Abstract, Introduction, Method sections (3.1–3.4), and Experiments (4), ensuring all technical claims are quoted or accurately represented.)

Improvements for AI systems

Based on a rigorous analysis of the VISD (Video Reasoning via Structured Self-Distillation) framework, I have identified three critical areas for improvement in current AI systems and their corresponding capabilities.

The core weakness in existing VideoLLMs is the coarse nature of reinforcement learning rewards (RLVR), which are sparse and sequence-level. VISD fundamentally transforms this by integrating a dedicated Video-Aware Judge Model (J) to generate structured, diagnostic feedback (f).

  • Mechanism: For every sampled trajectory, J evaluates the response across multiple dimensions:

  1. Answer Correctness (Ans): Is the final result accurate?

  2. Logical Consistency (Log): Does the reasoning process support the answer?

  3. Spatiotemporal Grounding (Time/Space): Are time stamps and bounding boxes correctly referenced?

  • Resulting Improvement: The AI system moves beyond simply knowing if it failed to knowing why. It identifies failure modes such as temporal misalignment or logical inconsistency, which previously required massive oversampling to diagnose.

The integration of self-distillation (dense supervision) and RL (sparse supervision) is notoriously unstable in existing systems, often leading to conflicting signals. VISD solves this via a Direction–Magnitude Decoupling strategy.

  • Mechanism:

  1. Direction Control (A): The rollout-level advantage, computed from the sparse environment rewards (R), dictates the fundamental direction of the policy update (the should we reinforce or suppress decision).

  2. Magnitude Modulation (t): The structured privileged information (t) modulates the magnitude of token-level updates. t is derived from a Top-K Local Support mechanism, ensuring the model compares its current prediction against a compact set of high-probability teacher candidates, including the realized token.

  3. Blending: The final token advantage (t) is a linear combination of these two elements: t = A(1-lambda) + lambda m t.

  • Resulting Improvement: The AI system achieves high-fidelity, fine-grained credit assignment. It receives the stability and task alignment of RL while gaining the dense, targeted corrections of self-distillation, without the optimization instability associated with mixing these signals naively.

Training long video sequences introduces high variance and instability in policy updates. VISD employs disciplined strategies to ensure robust learning over thousands of tokens.

  • Mechanism:

  1. Curriculum Scheduling (lambda Annealing): The system starts with a high influence of structured self-distillation (lambda 1), allowing the model to rapidly learn fine-grained reasoning patterns from the judge's guidance. As training progresses, lambda is annealed towards 0, smoothly transitioning the model to rely on real-world environment rewards (pure RL).

  2. EMA Teacher Stabilization: A continuously updated Exponential Moving Average (EMA) of the policy parameters is used to define the teacher model. This smooth update mitigates high-frequency fluctuations in the teacher's scoring, preventing unstable overfitting to instantaneous student performance.

  • Resulting Improvement: The AI system exhibits significantly faster convergence (up to 2 times faster in optimization steps) and maintains a stable learning trajectory across long-horizon video sequences, avoiding the pitfalls of early overconfidence or sudden catastrophic forgetting.

The integration of these mechanisms enables an improved VideoLLM to perform the following tasks with superior fidelity compared to current state-of-the-art models:

  1. Provide Verifiable Grounding: The system can generate reasoning that is not only correct but also demonstrably faithful to the video evidence, precisely localizing objects and events using correct temporal and spatial coordinates (e.g., correctly identifying that a panda is inside a bucket, not just near it).

  2. Interpret its Own Errors: When asked to explain its reasoning or why it failed, the system can diagnose its own failure modes (e.g., My initial hypothesis was logically inconsistent with the visual evidence at 154s) rather than providing a generic incorrect answer.

  3. Learn Efficiently: It achieves high performance in complex reasoning tasks using a substantially reduced number of training steps, making it highly sample-efficient and computationally superior to current baselines.

Sources

Related papers