How Well Do Vision-Language Models Understand Movies?

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "How Well Do Vision-Language Models Understand Movies?".

Jane: The gist: MF2 introduces a new benchmark for evaluating whether models can comprehend, consolidate,

Tom: First, who's behind it and why it matters.

Title and authors: Jane: So the paper is titled "How Well Do Vision-Language Models Understand Movies?" and it's written by a whole team of researchers from different places, including institutions in Lisbon, Amsterdam, Chapel Hill, and more >

Tom: Yeah. The title itself sets the stage for what they are testing: how well these vision-language models actually understand movies >

Lu: It shows a clear ambition to test if models can handle the complexity of long-form video understanding which is really a frontier right now >

Meng: I wonder if that's realistic for current systems. Training models to track an entire fifty minute movie story seems like a massive undertaking, especially when you have all these variables involved >

Lalam: The team is clearly pushing for this, focusing on the idea that real-world applications like education or storytelling depend on integrating and reasoning over information that unfolds over extended periods >

Tom: They are moving away from just seeing if a model can retrieve a piece of information and instead testing its ability to actually connect the dots across time >

The paper's summary: Jane: Okay, so what's this MF2 benchmark actually doing? The summary says they have constructed over five hundred thirty full-length, open-licensed movies, each paired with manual claim pairs >

Tom: They created over eight hundred and fifty of these claim pairs—one true fact and one plausible but false statement—targeting core narrative elements like character motivations and causal chains >

Lu: The key thing they emphasize is that they aren't using multiple-choice formats; they use a binary claim evaluation protocol where the model has to correctly identify both the true and the false claims >

Meng: That sounds tough for production. So instead of just saying "this answer is right or wrong," it’s forcing them to check two specific things at once >

Lalam: It’s meant to reduce biases, like when models might just pick an answer based on superficial cues, and instead force them to reason precisely about the relationship between the fact and the lie >

Tom: Right. And they are labeling these claims by granularity—single-scene, multi-scene, or global—to test how deep their understanding goes across different levels of narrative complexity >

The paper's improvements: Jane: I saw something in the summary about the improvements they suggest for this approach. It sounds like they are addressing some weaknesses in previous video benchmarks >

Tom: They explicitly state that this MF2 benchmark shifts away from just testing simple retrieval and instead focuses on testing actual narrative comprehension, which is what was missing before >

Lu: The main improvement is moving the focus toward causal relationships and emotional arcs between characters, which requires reasoning across different granularities like single-scene versus global understanding >

Meng: So they are making sure the task isn't just about identifying a character's name, but actually knowing *why* they did something or *how* one event led to another >

Lalam: They are also profiling this comprehension into specific dimensions like event and entity understanding, temporal perception, emotion understanding, and causal reasoning so you know exactly what aspect of narrative is being tested >

Tom: And there's a focus on how they handle input modalities. They test both video-only and video with subtitles to see which is better for getting the right answer >

Conclusion: Jane: So, wrapping up this discussion about "How Well Do Vision-Language Models Understand Movies?", what’s the big picture here? What does this mean for AI research moving forward?

Tom: The main point is that these models currently fall significantly short of human performance on long movie understanding, even with these improved evaluation methods >

Lu: They show that while there is progress in long context windows, the abstractive reasoning required for central storyline comprehension still isn't fully captured by current benchmarks >

Meng: The work suggests that relying solely on video input alone has limitations, and incorporating textual cues like subtitles or synopses seems to boost model accuracy substantially >

Lalam: It points toward a future where success in these areas depends on integrating and reasoning over information across extended periods, which is where we are heading >

Tom: So, the MF2 benchmark gives us a rigorous way to diagnose true narrative understanding by focusing on long-form movie narratives and binary claim evaluation >

Jane: It’s a really detailed setup. We're seeing that even with models like Gemini two point five Pro achieving high scores, they are still not at human level for this kind of deep narrative reasoning > <ref:2506.06275#pg3>

Tom: Incorporating transcripts significantly boosts model accuracy, and the framework effectively targets the harder challenge of global narrative understanding which current models continue to struggle with >

Lu: The limitation they mention is that annotators can’t fully control for memory effects when labeling those claims, so future work could involve splitting claims up and having different people rate them to isolate that effect >

Meng: For deployment, it means we need to be careful. If we rely on video alone, our accuracy drops because the models struggle with visual-only entity understanding >

Lalam: It’s a solid framework for evaluating what narrative comprehension actually means for vision-language models today.