VisionCoach: Reinforcing Grounded Video Reasoning via Visual-Perception Prompting

arXiv:2603.14659 · cs.CV, cs.AI · Submitted 2026-08-22 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "VisionCoach: Reinforcing Grounded Video Reasoning via Visual-Perception Prompting".

Jane: V ISION C OACH:

Tom: First, who's behind it and why it matters.

Title and authors: Tom: Now let's talk about who put this together. The paper is "VisionCoach: Reinforcing Grounded Video Reasoning via Visual-Perception Prompting," and it was written by Shoubin Yu, Yue Zhang, Mohit Bansal from the University of North Carolina at Chapel Hill. It’s interesting to see a team working on such a complex problem in video understanding.

Jane: Yes, those authors have built up some solid work in AI research before this paper came out, so we can expect a lot of technical depth here regarding their specific methods for grounding evidence in video sequences.

Lu: The collaboration between those researchers suggests they're tackling the challenge from multiple angles, which often leads to more robust solutions when dealing with complex domains like video reasoning.

Meng: I'm curious if their background in different areas helps them design a framework that doesn't just work on one type of data but is adaptable to various kinds of visual information.

Lalam: The fact that they are focusing specifically on "visual-perception prompting" suggests they are prioritizing the quality and relevance of the visual input during training, which is crucial for making sure the model learns what actually matters in a video.

The paper's summary: Tom: So, to sum up what this paper is about, "VisionCoach: Reinforcing Grounded Video Reasoning via Visual-Perception Prompting" proposes an input-adaptive reinforcement learning framework designed to boost how well models can find and track important evidence within videos.

Jane: Essentially, they've noticed that when the model struggles with spatio-temporal grounding—meaning it can’t reliably locate what is happening across different times and places in a video—existing methods often fall apart, either by relying on language hints or by using tools that make things slow down way too much.

Lu: They tackle this by introducing a mechanism where visual prompts are selectively added to challenging inputs during the training phase to help the model pick up on relevant evidence while ignoring distracting elements.

Meng: That sounds like they're trying to teach the model *how* to look for specific things in a video, instead of just hoping it finds them randomly by looking at everything equally.

Lalam: And then, they use self-distillation to make sure the model internalizes that improved behavior so that when it's actually used later, like at inference time, it can reason directly on raw videos without needing those extra prompts.

The paper's improvements: Tom: One of the key improvements they highlight is their use of a specialized object-aware spatial grounding reward during reinforcement learning training. This reward isn't just about whether an object was found, it specifically enforces identity consistency and checks for multi-region bounding-box overlap.

Jane: That’s significant because it moves beyond simple success or failure; they are making the model accountable for keeping track of the same object throughout a sequence, which is exactly what video reasoning demands.

Lu: By enforcing that object identity consistency across frames and checking spatial overlap with ground truth, they are directly addressing the issue of hallucinated explanations driven by language priors because the visual evidence must match what's actually there.

Meng: From my side, having those specific constraints on bounding-box overlap makes sense for practical applications where you need to know exactly where a component is located relative to others in a scene.

Lalam: And the self-distillation step really solidifies this; it ensures that the model doesn't just get the reward during training but truly internalizes that grounded perception behavior so it can operate efficiently later on without external visual prompting.

Conclusion: Tom: So to wrap up on "VisionCoach: Reinforcing Grounded Video Reasoning via Visual-Perception Prompting," the authors have presented an input-adaptive RL framework that uses training-time visual prompting and self-distillation to get better spatio-temporal grounding. They showed it performs very well, surpassing GPT-4o on the V-STAR benchmark and showing strong results across several general video understanding tasks.

Jane: It really shows that guiding the model's perception during training with targeted visual cues can lead to a more reliable way for AI to understand complex video sequences without needing those extra, slow perception tools at inference time.

Lu: The implication is that we might see models become much better at understanding long-form video content where tracking objects and events across extended periods is necessary for accurate answers.

Meng: For practical deployment, the ability to reason directly on raw videos without a separate cropping or tool-calling step simplifies the architecture significantly and makes it much more usable in real-world systems.

Lalam: I'm really optimistic about this direction; if we can internalize these grounding improvements this way, it could lead to AI that is far more capable of nuanced, multi-step understanding in video tasks.

Shoubin Yu, Yue Zhang, Mohit Bansal, Daeun Lee

University of North Carolina, Chapel Hill

cs.CV, cs.AI

Submitted: 2026-08-22

Updated: 2026-08-25

Comments: Accepted to ECCV 2026; Project website: https://visioncoach.github.io/

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 94/100

The gist: V ISION C OACH: Reinforcing Grounded Video Reasoning via Visual-Perception Prompting The paper addresses the challenge in video reasoning where existing approaches "still struggle to achieve reliable

Key concepts

VisionCoach
A proposed input-adaptive reinforcement learning framework designed to boost how well models can find and track important evidence within videos by selectively adding visual prompts during training.
Spatio-temporal grounding
The ability of a model to reliably locate what is happening across different times and places within a video sequence. Existing methods often struggle with this aspect.
Self-distillation
A technique used to ensure the model internalizes improved grounded perception behavior during training, so it can operate efficiently later without needing external visual prompting at inference time.

Terminology

Summary

V ISION C OACH: Reinforcing Grounded Video Reasoning via Visual-Perception Prompting

The paper addresses the challenge in video reasoning where existing approaches still struggle to achieve reliable spatio-temporal grounding during the reasoning process. Current methods are either text-centric and often generate hallucinated explanations driven by language priors rather than faithful visual observations, or they rely on external perception tools... which introduce additional computational overhead due to repeated tool invocation and multi-stage processing during inference.

To address this, the authors propose V ISION C OACH, an input-adaptive RL framework that improves spatio-temporal grounding through visual prompting as training-time guidance. The core methodology involves selectively applying visual prompts to challenging inputs to amplify question-relevant evidence and suppress distractors during the Reinforcement Learning (RL) phase. Crucially, the model then internalizes these improvements through self-distillation, enabling grounded reasoning directly on raw videos without visual prompting at inference.

V ISION C OACH is composed of two primary components:

  1. Visual Prompt Selector (VP-S ELECTOR): This component predict[s] appropriate prompt types conditioned on the video and question to provide adaptive perception guidance for challenging inputs. The VP-S ELECTOR is trained using a proxy reasoner dataset, where it is optimized with SFT to select the most effective visual prompt type.

  2. Spatio-Temporal Reasoner (ST-R EASONER): This component performs grounded reasoning under visual prompt guidance and is optimized via RL using specific rewards.

The training of the ST-R EASONER incorporates a specialized object-aware spatial grounding reward. This reward is designed to enforce object identity consistency and multi-region bounding-box overlap, which encourages accurate multi-object grounding and temporally consistent reasoning.

Training Pipeline:

The overall training procedure involves identifying hard samples based on initial low rewards. For these challenging inputs, the VP-S ELECTOR generates an optimal visual prompt (v i), creating a visual-prompted input x'i. The ST-R EASONER then performs rollouts using this prompted input. To ensure the model internalizes this behavior without needing prompts at inference, self-distillation is applied. This process selects the top N rollouts whose rewards exceed the average reward of the original rollouts and applies token-level negative log-likelihood (NLL) to these selected reasoning trajectories.

Inference:

The final system maintains a single efficient inference pathway, allowing the model to perform reasoning directly on raw videos without visual prompting at inference.

Evaluation and Results:

V ISION C OACH was evaluated across several benchmarks, including V-STAR, VideoMME, WorldSense, VideoMMMU, PerceptionTest, and Charades-STA. The results demonstrate that V ISION C OACH achieves state-of-the-art performance under comparable settings. Specifically:

  • On the V-STAR benchmark (spatio-temporal reasoning), V ISION C OACH surpasses GPT-4o and improves over Qwen2.5-VL-7B by +15.0% mAM and +25.1% mLGM, establishing new state-of-the-art performance.

  • In general video understanding and temporal grounding benchmarks, it consistently outperforms prior open-source VideoLLMs.

Key Contributions:

The authors summarize their contributions as:

  • They propose an input-adaptive RL framework for video reasoning that explicitly guides spatiotemporal grounding through training-time visual prompting and self-distillation.

  • They design an object-aware spatial grounding reward that incorporates object identity consistency and multi-region bounding-box IoU to better support spatial-grounded reasoning.

  • They introduce a visual prompt selector with a proxy-reasoner-based data construction pipeline to predict appropriate visual prompts for video QA inputs.

  • Finally, they demonstrate that V ISION C OACH achieves SoTA performance across a wide range of video reasoning, video understanding, and temporal grounding benchmarks.

Improvements for AI systems

Based on a rigorous analysis of the V ISION C OACH framework, I have identified several critical architectural and methodological improvements that can be generalized beyond video reasoning to enhance complex AI systems. These improvements address the fundamental weakness in current multi-step reasoning models: their inability to reliably ground intermediate steps without external intervention or hallucination.

The following are specific, actionable enhancements for an advanced AI system:

A. Implementation of an Input-Adaptive Guidance Selector (VP-S ELECTOR)

Instead of relying on static or hardcoded prompts, the system should integrate a lightweight predictor that determines the optimal guiding mechanism based on the input context.

  • Action: Train a dedicated sub-module (e.g, using LoRA fine-tuning) to map an input state (e.g., a complex document chunk or a specific reasoning problem) to one of several predefined guidance strategies (e.g, Focus on Section X, Verify this Value, Trace the Dependency).

  • General Application: This allows the system to dynamically select the most effective form of perceptual or logical guidance, rather than applying a generic instruction.

B. Integration of Object/Entity-Aware Reward Constraints (r spa)

The reinforcement learning (RL) reward function must be augmented to enforce consistency across multiple reasoning steps, moving beyond simple binary success/failure metrics.

  • Action: For any multi-entity or complex event, the reward should not only check if an entity was found but also verify:

  1. Identity Consistency: That the perceived identity of (e.g.) The CFO remains consistent across all frames/steps where they appear.

  2. Multi-Region Overlap (IoU): That the spatial or temporal evidence provided by the model overlaps significantly with ground-truth annotations, even if multiple objects are involved in a single action.

  • ** General Application:** This prevents models from hallucinating plausible but incorrect entities or mislocating critical information during complex tasks like scientific process tracing or legal document analysis.

C. Adoption of Guided RL with Self-Distillation (LSD) The most significant improvement is the transition from requiring external guidance at inference to internalizing that guidance during training.

  • Action:

  1. Hard Sample Identification: Use the current policy's low-reward outputs to identify hard samples where grounding fails.

  2. Targeted Prompting: Apply the VP-S ELECTOR's recommended guidance (v i) to these hard samples, creating a guided input (x'i).

  3. RL Optimization: Train the policy model (ST-R EASONER) using this guided input and grounded rewards.

  4. Self-Distillation: Crucially, use the high-reward trajectories generated under guidance as a teacher to fine-tune the original, unguided policy model (pi theta).

  • ** General Application:** This process allows the core model to internalize successful grounding behaviors. The system learns why a piece of evidence is relevant, rather than just being told that a path leads to success.

By implementing these changes, the improved AI system will achieve the following specific capabilities:

  1. Reliable Spatio-Temporal/Entity Grounding: The system will consistently locate and cite specific evidence (e.g., The blue component at 45s) rather than relying on vague language priors (A blue thing was involved).

  2. Reduced Inference Overhead: Unlike tool-based systems (which require external API calls, RAG lookups, or cropping), the improved system operates with a single, efficient forward pass at inference time because the necessary grounding knowledge has been internalized via self-distillation.

  3. Adaptive Problem Solving: The system can identify when its current reasoning path is failing (a hard sample) and autonomously apply an adaptive intervention (the VP-S ELECTOR guidance) to correct its trajectory, without requiring human intervention or a separate tool call.

  4. Consistent Multi-Step Reasoning: The object-aware rewards ensure that even in long, complex sequences, the identity of key entities remains consistent across all steps, preventing the model from accidentally changing a white sedan into an orange car mid-reasoning.

Sources

Related papers