Tracking the Truth: Object-Centric Spatio-Temporal Monitoring for Video Large Language Models

arXiv:2605.08974 · cs.CV, cs.AI · Submitted 2026-08-15 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Tracking the Truth: Object-Centric Spatio-Temporal Monitoring for Video Large Language Models".

Jane: The paper was written by Tri Cao, Khoi Le, Thong Nguyen, Cong-Duy Nguyen, Quynh Vo et al. from National University of Singapore and VinUniversity and Nanyang Technological University.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Title: Tom: Welcome back to the show, everyone. Today we're digging into a paper that's got a wonderfully ambitious title: "Tracking the Truth: Object-Centric Spatio-Temporal Monitoring for Video Large Language Models." Jane, I gotta say, just reading that title gets me excited.

Jane: It really does, Tom. And that title is doing a lot of heavy lifting. It's telling us that these video AI models, the ones that watch footage and answer questions about it, they have a serious problem. They hallucinate. They make things up that just aren't in the video.

Tom: Right, and the authors argue the root cause is that these models aren't actually tracking what's happening over time. They're not monitoring objects, their identities, or their states as the video plays. So the title is basically a promise to fix that.

Jane: Exactly. Think of it like watching a soccer game. A human viewer keeps track of player number sixty-six even when he's in a crowd or moving fast. We know he's wearing a red shirt, and we remember he passed the ball to number twenty-seven. These AI models, they often lose that thread. They see a flash of red and a goal, and they just guess.

Tom: And that's why the authors built a new benchmark called STEMO-Bench. It's designed to catch these models in the act of guessing. Instead of just asking a final question like "Did player sixty-six pass to twenty-seven before the goal?", they break it down into tiny sub-questions.

Jane: It's like a teacher asking a student to show their work on a math problem. You can't just write down the answer. You have to prove you understand each step. That's the core idea here, and it's a really clever way to expose when a model is being lazy and just using a statistical shortcut.

Tom: So the title isn't just catchy. It's a direct challenge to the entire field of video understanding. It's saying, "Hey, you might be getting the right answers, but you're not actually tracking the truth."

Summary: Jane: So we've established that "Tracking the Truth" is about holding video AI models accountable. Let's talk about what the paper actually found when they put these models to the test.

Tom: And the results are pretty damning, Jane. They tested a bunch of top-tier models, including some really powerful proprietary ones like Gemini and GPT-five. On their new benchmark, the best end-to-end model only got about sixty-one percent of the target questions right. That's barely better than a coin flip on a two-option question.

Jane: But the really interesting part is the consistency metric. The paper introduces something called "Consistency," which measures whether a model gets the final answer right *and* correctly answers all the little sub-questions that support it. And that's where things fall apart.

Tom: Right. A model might get the final answer correct, but then fail the sub-question about the player's shirt color. That means it got lucky, not smart. The authors call this "coincidental correctness." It's like a student guessing "forty-two" on a test and getting it right, but not being able to explain why.

Jane: And that's the whole point of the paper's summary. The authors are showing that our current way of evaluating video AI is fundamentally flawed. We're just looking at the final answer, but we're ignoring the reasoning process. This paper proves that a lot of these models are essentially bluffing their way through video understanding.

Tom: They also found that these models struggle with what they call "identity drift." An object might change slightly, like a player turning around, and the model forgets it's the same person. It's a fundamental failure of spatio-temporal monitoring.

Jane: So the summary is essentially a wake-up call. It's saying that the field has been celebrating progress that might not be real. The models are good at pattern matching, but they're not truly understanding the dynamic world in front of them. That's a huge deal.

Tom: And it sets the stage perfectly for the solution they propose, which we're going to get into next. But for now, let's just say the diagnosis is clear: video AI has a truth-tracking problem.

Improvements: Jane: We've heard about the problem, so now let's talk about the fix. The paper doesn't just point out flaws; it introduces a whole new framework called STEMO-Track. Tom, what's the big idea here?

Tom: The big idea, Jane, is to stop treating a video like one giant blur of pixels. Instead, STEMO-Track forces the AI to build a structured, object-centric representation of the world. It's like giving the AI a notebook to write down what it sees.

Jane: And how does that notebook work in practice?

Tom: First, the video is chopped into chunks. Then, a "state extractor" looks at each chunk and identifies the objects and their states. So it might note, "Player sixty-six is wearing a red shirt," or "The ball is with Player twenty-seven." These are local observations.

Jane: But a notebook with random notes isn't enough. You need to connect them over time.

Tom: Exactly. That's where the "temporal aggregation" step comes in. This module links those observations across the chunks. It figures out that the "red shirt player" in chunk one is the same as the "player sixty-six" in chunk three. It builds a continuous trajectory for each object.

Jane: So instead of a bunch of disconnected snapshots, you get a story. "Player sixty-six wearing red, passes the ball to Player twenty-seven who then scores." That's a trajectory.

Tom: Precisely. And then, when you ask the AI a question, it doesn't go back to the raw video. It retrieves the relevant trajectories from its notebook and reasons over that structured information. It's a much cleaner, more reliable process.

Jane: And the results show it works. The paper reports that STEMO-Track with a Gemini-three-Flash backbone jumped to seventy-seven percent target accuracy, a huge leap over the sixty-one percent from the best end-to-end model. More importantly, its consistency score was much higher, meaning it was getting the answers right for the right reasons.

Tom: The improvements are clear. By forcing the model to explicitly track objects and their states over time, we get a massive boost in both accuracy and faithfulness. It's a really elegant solution to a really messy problem.

Conclusion: Tom: Alright, Jane, we've covered a lot of ground on "Tracking the Truth: Object-Centric Spatio-Temporal Monitoring for Video Large Language Models." Let's wrap this up.

Jane: Let's do it. We started by talking about the problem of hallucination in video AI, and how this paper diagnoses it as a failure of spatio-temporal monitoring.

Tom: Then we saw their benchmark, STEMO-Bench, which exposes these failures by breaking questions down into sub-questions. It's a much more honest way to test these models.

Jane: And finally, we looked at their solution, STEMO-Track, which builds explicit object trajectories and reasons over them. It's a clear win, showing significant improvements in accuracy and consistency.

Tom: The takeaway here is that we can't just trust a final answer from a video AI. We need to see the work. This paper gives us the tools to do that, and it points the way toward more reliable, more truthful video understanding.

Jane: It's a fantastic piece of research, and it's going to be a benchmark for the field. We're sad to say goodbye to this paper, but we're excited to see what comes next.

Tom: Thanks for listening, everyone. We'll be back soon with another paper to break down. Until then, keep asking the hard questions.

Tri Cao, Khoi Le, Thong Nguyen, Cong-Duy Nguyen, Quynh Vo, Anh Tuan Luu, Chunyan Miao, See-Kiong Ng, Shuicheng Yan, Bryan Hooi

National University of Singapore · VinUniversity · Nanyang Technological University

cs.CV, cs.AI

Submitted: 2026-08-15

Updated: 2026-08-18

Comments: Code: https://github.com/nguyentthong/video_hallucination

Code: https://github.com/nguyentthong/video_hallucination

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: The authors argue that while MLLMs have advanced video understanding, they remain "highly prone to hallucinations in dynamic scenes." They attribute this to a failure in "spatio-temporal monitoring,

Key concepts

Hallucination
When video AI models make up information that is not present in the actual video footage. The hosts argue this happens because the models are not properly tracking objects or their states as the video plays, leading to incorrect guesses instead of factual understanding.
STEMO-Bench
A new benchmark created by the authors to test video AI models. Instead of asking one final question, it breaks down complex questions into many small sub-questions, requiring models to prove their reasoning for each step.
STEMO-Track
A proposed solution that forces AI to build a structured, object-centric representation of the world. It involves chopping video into chunks, identifying objects and states in those chunks, and then linking these observations over time to create continuous object trajectories.
Consistency Metric
A metric introduced by the paper that measures whether a model gets the final answer right AND correctly answers all the supporting sub-questions. A high consistency score indicates the model is getting the correct answers for good reasons, not just by luck.

Terminology

Summary

Summary

The paper introduces STEMO-Bench (Spatio-TEmporal MOnitoring), a benchmark designed to evaluate object-centric spatio-temporal monitoring in video Large Language Models (MLLMs), and STEMO-Track, an object-centric framework for video understanding.

Problem Statement: The authors argue that while MLLMs have advanced video understanding, they remain highly prone to hallucinations in dynamic scenes. They attribute this to a failure in spatio-temporal monitoring, the ability to persistently track object identities, states, and relations over time. Existing benchmarks obscure this deficit by relying on single final-answer evaluations for queries that can often be resolved via local visual cues or statistical priors.

STEMO-Bench Construction: The benchmark uses a structure-first data construction pipeline. Annotators construct human-verified structural representation of each video, explicitly capturing objects, their evolving states, and temporal relations. The pipeline involves: (1) Temporal State Anchoring, (2) Entity Registration, (3) Fact Extraction, and (4) Cross-Time Identity Linking. An LLM then generates depth-bounded reasoning chains over object trajectories to create target questions, which are decomposed into atomic, supporting sub-questions mapped directly to the facts. Human annotators verify that the language is clear and natural, the sub-questions are jointly necessary and sufficient to answer the target query, and all answers are unambiguously grounded in visual evidence. The dataset contains 88 curated videos across sports, egocentric, surveillance, and instructional domains, with 977 target questions and 2516 supporting sub-questions (3.25 per target on average). The videos average 183 seconds.

Evaluation Metrics: The paper defines three metrics: Target Accuracy (Atarget) for complex questions, Sub-question Accuracy (Asub) for foundational perception, and Consistency (Acons) which Evaluates whether the model maintains coherent intermediate reasoning when the final target prediction is correct. The authors note that A model with high Atarget but low Acons is likely relying on spurious correlations rather than maintaining a robust spatio-temporal monitor.

STEMO-Track Framework: The proposed framework first converts a video into structured object trajectories and then performs question answering over this representation. It consists of: (1) Chunk-wise State Extraction, where a VLM-based state extractor processes temporal chunks to identify objects and states; (2) Temporal Aggregation, which links observations across chunks using similarity scoring, constraint filtering, conflict resolution, and trajectory generation to form coherent object trajectories; (3) Query-based Trajectory Retrieval, selecting relevant trajectories; and (4) Trajectory-Guided Answering, generating answers from retrieved trajectories. The representation is constructed once per video and reused for downstream tasks.

Main Results: STEMO-Track (Gemini-3-Flash) achieves 77.1% Atarget, 76.3% Asub, and 81.3% Acons, outperforming Gemini-3.1-Pro by 15.7, 11.4, and 12.1 points, respectively. The authors report "statistically significant (p < 0.005) improvements via paired bootstrap tests. The framework also transfers successfully to established video benchmarks" including VideoHallucer, Video-MME, and EgoSchema.

Ablation Studies: Key findings include: lowering the answerer visual budget to 8 frames barely impacts Gemini-3-Flash, as trajectory memory encodes key temporal cues, but severely degrades Qwen3-VL; removing text trajectories or visual frames degrades performance; 15-second chunks provide optimal balance; structured JSON representation outperforms prose summaries; and shuffling the temporal order of extracted states yields a severe performance drop to 65.3%, underscoring that maintaining the correct temporal sequence is paramount.

Qualitative Analysis: The paper demonstrates that baselines often pass target questions while failing prerequisite sub-questions, showing that target accuracy alone overestimates true video understanding. The trajectory representation corrects baseline failures in temporal tracking.

Conclusion: The authors state that explicit trajectory construction substantially improves spatio-temporal reasoning consistency and reduces hallucinated predictions. Future work will focus on improving fine-grained visual attribute binding within object trajectories.

Improvements for AI systems

Based on the paper, here are the specific improvements I can implement and the resulting capabilities of the improved AI system:

Implementation: Replace monolithic video encoding with a two-stage pipeline:

  • Stage 1 (State Extraction): Partition video into 15-second chunks, sample 60 frames per chunk, and use a VLM (e.g., Gemini-3-Flash or Qwen3-VL) to extract structured JSON states: object id, timestamp, attribute, relation.

  • Stage 2 (Temporal Aggregation): Link object observations across chunks using identity-linking with constraints (temporal proximity ≤15s, confidence threshold ≥0.75, bipartite matching for conflict resolution). Store as persistent object trajectories in a structured memory.

  • Stage 3 (Query-Guided Retrieval): For each question, retrieve only relevant trajectories (filter by object, state, temporal interval) before generating the answer.

Resulting Capability: The system now explicitly tracks object identities, state changes, and temporal orderings over long videos (up to 3+ minutes) without losing track during occlusions or state transitions. It can answer questions like Did player #66 pass to #27 before scoring? by retrieving the exact trajectory sequence rather than guessing from frame-level patterns.

Implementation: Integrate a sub-question decomposition layer into the AI system's inference:

  • Before answering a complex target question, automatically generate atomic sub-questions (e.g., Does player #66 wear red?, Does #66 pass to #27?, Does #27 score?).

  • Require the system to answer all sub-questions first, then synthesize the final answer only if all prerequisites are satisfied.

  • Track consistency: if the final answer is correct but sub-questions are wrong, flag as coincidental correctness and suppress the answer.

Implementation: Add a post-processing layer that checks for temporal contradictions:

  • After generating an answer, validate it against the object trajectory memory.

  • If the answer references an event order (e.g., before, after), verify the timestamps in the trajectories.

  • If the answer attributes a state to the wrong object (e.g., red shirt girl vs. blue shirt girl), correct it using identity-linking data.

Implementation: Dynamically adjust frame sampling based on query complexity:

  • For simple queries (e.g., Is there a red car?), use only 8 frames at the answerer stage.

  • For complex multi-object temporal queries (e.g., Did A pass to B before C scored?), use full 64-frame context plus trajectory memory.

  • Use the trajectory memory as a fallback when visual frames are insufficient (e.g., occlusion).

Implementation: Replace free-form prose descriptions with structured JSON tuples for all extracted states:

  • Format: "object": "player 66", "time": "t1", "attribute": "wearing red", "relation": "passes to", "target": "player 27"

  • Use this structured format consistently across extraction, aggregation, retrieval, and answer generation.

  1. Long-Form Video QA with Verified Reasoning: Answer complex questions about 3+ minute videos with 77.1% accuracy, while providing step-by-step verification of each reasoning step (81.3% consistency).

  2. Identity-Preserving Tracking: Follow objects through occlusions, state changes, and interactions without confusing them (e.g., correctly distinguishing player #66 from player #27 even when both wear red).

  3. Temporal Order Verification: Explicitly verify event sequences (e.g., X happened before Y) by referencing timestamped trajectories, eliminating temporal hallucinations.

  4. Hallucination Suppression: Detect and reject answers that lack supporting evidence in the trajectory memory, reducing false positives by 15.7% compared to frontier models like Gemini-3.1-Pro.

  5. Efficient Resource Utilization: Achieve near-peak performance with only 8 frames at the answerer stage (76.5% accuracy), making it suitable for real-time applications.

  6. Cross-Benchmark Generalization: Transfer improvements to standard benchmarks (VideoHallucer: 83.5%, Video-MME: 79.9%, EgoSchema: 66.2%), outperforming both proprietary and open-weight baselines.

  7. Explainable Outputs: Provide structured reasoning chains (sub-question answers + trajectory references) alongside final answers, enabling human auditing and trust in high-stakes applications (e.g., surveillance, sports analytics, medical video review).

Bottom Line: The improved system replaces implicit, monolithic video understanding with explicit, structured, object-centric reasoning. It tracks entities persistently, verifies temporal orderings, and refuses to answer when evidence is insufficient—resulting in measurably fewer hallucinations and more trustworthy outputs for real-world video understanding tasks.

Abstract

While multimodal large language models (MLLMs) have advanced video understanding, they remain highly prone to hallucinations in dynamic scenes. We argue this stems from a failure in spatio-temporal monitoring, the ability to persistently track object identities, states, and relations over time. Existing benchmarks obscure this deficit by relying on single final-answer evaluations for queries that can often be resolved via local visual cues or statistical priors. To rigorously diagnose this, we introduce STEMO-Bench (Spatio-TEmporal MOnitoring), a benchmark of human-verified object-centric facts that evaluates intermediate reasoning by decomposing queries into sub-questions, distinguishing genuine temporal understanding from coincidental correctness. To address failure modes exposed by STEMO, we propose STEMO-Track, a novel object-centric framework that explicitly constructs and reasons over structured object trajectories via chunk-wise state extraction and temporal aggregation. Extensive experiments demonstrate that our object-centric framework significantly reduces hallucinated answers and improves spatio-temporal reasoning consistency over state-of-the-art MLLMs.

Sources

Related papers