2608.07417-I Seek You in Videos: Identity-Conditioned Queries for Person-Centric Video Reasoning

page_by_page

Video file (mp4)

In short

The episode discusses a paper introducing ISYV, a benchmark and model for person-centric video reasoning, where a model tracks a specific person across videos using a reference image. The hosts highlight that human accuracy is 95%, while the best closed-source model (Gemini 2.5 Pro) reaches 67%, and their 7B model achieves 57% after reinforcement fine-tuning. They also explore the dataset pipeline, cognitive difficulty levels, and failure modes like answer hacking.

Key concepts

Identity-conditioned Queries (ICQ)
A task where a model receives a reference image of a person and a video, then answers questions about that specific person's actions, interactions, and intentions, even across outfit changes and scene transitions. It requires integrating video, image, and text inputs simultaneously.
Answer hacking
A failure mode where a model ignores the reference image and guesses answers based only on the video and question. The paper found some models do this almost a third of the time, so they created a separate metric (ICQ QA accuracy) that only counts answers showing engagement with the reference image.
Effective Shot Re-reasoning (ESR) reward
A reinforcement learning reward that encourages the model to identify which shots in a video are relevant to the target person, without needing ground-truth shot annotations. This allows the model to learn shot selection through training alone, avoiding expensive manual annotation.
Cognitive hierarchy
The benchmark organizes questions into six levels of difficulty, mirroring human cognitive development, from basic perception to understanding hidden intent and causation. This helps researchers see exactly where models break down, as human accuracy is over 95% but models lag significantly.

This episode discusses

Transcript

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "I Seek You in Videos: Identity-Conditioned Queries for Person-Centric Video Reasoning".

Jane: The paper was written by Shibo Gao, Chongxiao Wang, Chenglong Huang, Jie Ma, Haolin Shi et al. from Beijing Jiaotong University and HUJING Digital Media & Entertainment Group and MAIS, Institute of Automation, Chinese Academy of Sciences and School of Computer Science and Technology, University of Chinese Academy of Sciences and College of Electronic and Information Engineering, Tongji University and School of Computing and Communications, Lancaster University.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper summary — Tom, Jane, Lu, Meng and Lalam discuss the paper 'I Seek You in Videos: Identity-Conditioned Queries for Person-Centric Video Reasoning' — the thesis, the key findings and why it matters.: Tom: So we've got a really interesting one today, a paper that basically asks a model to find one specific person in a whole video, even when that person changes clothes, moves to different scenes, or disappears for a while.

Jane: And it's not just finding them, right? The model has to answer questions about what that person is doing, who they're talking to, why they're doing it, all while keeping track of that one identity across all the chaos of a real video.

Lu: The setup is clever. You give the model a reference image, a still photo of a person, and then a long video with all these shot transitions, and the model has to reason about that specific person in the video.

Meng: What struck me is how they built this to mirror human cognitive development, six levels of difficulty that go from basic perception all the way up to understanding hidden intent and causation.

Tom: That's the part that really got me. They didn't just make another video QA benchmark, they organized it like a cognitive hierarchy, so you can see exactly where models break down.

Jane: And the numbers tell a pretty stark story. Human accuracy is over 95 percent, but the best closed-source model, Gemini-2 point 5-Pro, only hits 67 percent. Open-source models mostly stay below 40 percent.

Lalam: Their own model, a 7B parameter system, reaches 57 percent after reinforcement fine-tuning. That's not quite closed-source level, but it's remarkably close given the size difference.

Lu: What's fascinating to me is the specific failure modes they uncovered, models losing track of a person across outfit changes, confusing the reference image with the last frame of the video, or just ignoring the reference image entirely.

Meng: That last one is called answer hacking in the paper. The model skips the image altogether, guesses based on the video and question alone, and some models do that almost a third of the time.

Lu: And that's why they had to design a separate metric, ICQ QA accuracy, which only counts answers that actually demonstrate the model engaged with the reference image.

Tom: The paper proposes a full package, a benchmark, a training dataset, plus a model architecture with a specialized module for compressing that reference image into something the model can actually use.

Jane: But let's be careful here, because the real innovation in the training is the reward function. They reward the model for correctly identifying which shots matter, without ever giving it ground-truth shot annotations.

Lalam: Which is a genuinely useful trick, because annotating which shots are relevant in a long video is expensive and subjective, but the model can learn it through reinforcement alone.

Meng: So we've got a task definition, a benchmark, a training set, a model, and a training strategy, all wrapped up in one package. I think it's fair to say this one has legs.

Tom: Alright, let's get into the paper itself. Page one is all about setting up why the old video reasoning paradigm just doesn't cut it anymore, and honestly, the opening argument is pretty convincing.

Page 1 of the paper — Discuss page 1 of the paper: what is new on this page, explained in simple terms. Do not repeat the thesis already covered; build on it.: Tom: So we've set the stage at a high level. Page one is really where they make the case that standard video reasoning has been stuck in this simple video-in, text-out paradigm.

Jane: And the real-world problem they're pointing at is that users don't just have a video and a question. They have a reference image, a photo of someone they care about, and they want to find that person.

Lu: The example in the paper is film production. An editor might have a photo of an actor and need to locate every scene where that actor appears, or track how their performance evolves across different takes.

Meng: And it's not just film, you can imagine surveillance, personalized analytics, video retrieval, all these applications where you have a specific identity and you need to reason about them.

Tom: They list three concrete challenges on this page. First, you have to integrate heterogeneous inputs, meaning a video plus an image plus a text question, all at once.

Jane: Second, you have to localize and track the target person through complex scenes, which usually means losing them at some point and picking them back up.

Lu: And third is the one I keep coming back to, cross-domain identity matching. The reference photo might have completely different lighting, clothing, even makeup compared to the video.

Meng: They call the whole thing the ICQ task, short for Identity-conditioned Queries, and they position it as filling this gap between simplified benchmarks and what people actually need.

Tom: What I appreciate is that they're upfront about the limitations of prior work. They cite image-level personalization systems like IDA-VLM and PLVM, but those only handle the image-text setting.

Jane: Right, those systems let you say "show me this person in this picture," but they never extend to video with all its temporal complications.

Lu: They also borrowed the idea of anchoring the task to human cognitive abilities, and I think that's a genuinely thoughtful design choice, because it gives researchers a framework for understanding which capabilities are missing.

Meng: And then they front-load their headline results right there on page one, the best closed-source model at 67 percent, open-source models below 40, and their own 7B model hitting 57 after training.

Jane: That last number is the one that made me do a double take. Most people assume you need a giant proprietary model to get anywhere near frontier performance, and they're showing that's not necessarily true.

Tom: They also mention the ESR reward in the abstract, which is the effective shot re-reasoning mechanism, and that's going to turn out to be a central piece of the whole framework.

Lu: I like that they save the explanation for later, though. Page one is really about painting the problem and giving you the headline numbers to make you want to read further.

Meng: But there's a subtle detail in their contribution list too. They claim the 7B model "rivals" closed-source models, and that phrasing is careful, because it doesn't quite match them.

Tom: Right, it rivals them but doesn't beat them, and the gap analysis across the different difficulty levels is going to be really informative when we get to the experiments section.

Jane: Before we jump there, page two covers the related work, and I think the way they position their benchmark against all the existing ones is worth digging into.

Page 2 of the paper — Discuss page 2 of the paper: what is new on this page, explained in simple terms. Do not repeat the thesis already covered; build on it.: Jane: Alright, page two dives into related work, and the key point is that all the major video reasoning benchmarks, TVQA, MVBench, LongVideoBench, Video-MME, they all use that same video-text dual-modal format.

Tom: And they all miss the multi-input aspect. None of them give you a reference image to condition on, and almost none of them really stress complex shot transitions.

Lu: The table on page four actually makes this visual. ISYV-Benchmark is the only row where you see both the multitask and multi-input columns marked as yes.

Meng: And the cognitive hierarchy column too, that's another exclusive. They chart it out with human-aligned cognition as a column header, and it's blank for every existing benchmark.

Tom: There's also a distinction they draw with domain-specific works like video anomaly detection and long-term multi-object tracking, where you're following specific subjects over time.

Jane: But those operate in fixed label spaces, you're detecting known anomaly categories, not doing open-ended reasoning about an individual's intentions or motivations.

Lu: I noticed they mention their own prior work in that area, including that pig tracking paper from ICIG 2023. That's a fun detail, group-housed pig tracking as a precursor to person-centric reasoning.

Meng: The RL fine-tuning section is where it gets technical. They trace the lineage from GRPO through DAPO, SRPO, GFPO, and Multi-GRPO, and point out that none of these have been validated on this kind of task.

Jane: And that's exactly the gap they're filling. They're taking these reinforcement learning algorithms that work for math and general reasoning and applying them to identity-conditioned video understanding.

Tom: The important distinction here is between frameworks like Video-R1, which reinforces video reasoning generally, versus what ISYV does, which is conditioning everything on a specific person reference.

Lu: And notice they cite their own Multi-GRPO work in that list, so they're building on their own contributions as well as the broader community.

Meng: I also think it's worth noting they mention model compression and quantization as complementary lines of work. That feels a bit out of place in a paper about reasoning, but it shows they're thinking about deployment costs.

Tom: Right, because if you're building a system that needs to handle personal video analytics, you can't always afford a giant API call to a proprietary model, you want something that runs locally.

Jane: That deployment angle connects back to the 7B parameter choice. They're deliberately targeting a size that's practical, not just chasing benchmark scores.

Lu: So page two gives us the landscape, everyone else is doing video-text, and the ones that do personalization are stuck at image level. The gap is clear.

Tom: Which sets us up perfectly for page three, where they start describing how they actually built the training set. And that's where things get surprisingly clever.

Page 3 of the paper — Discuss page 3 of the paper: what is new on this page, explained in simple terms. Do not repeat the thesis already covered; build on it.: Tom: Page three is where the dataset construction starts, and the first thing that jumps out is the scale. They collected over 100,000 video clips from films, TV series, and anime.

Jane: And that's before filtering. After all the quality checks, they end up with 24,150 unique clips and 74,578 QA pairs for training, and separately 1,377 samples for the benchmark.

Lu: The construction pipeline is the real story here, though. They use TransNet V2 to segment each video into shots with timestamps, then Gemini-2 point 5-Pro generates detailed descriptions for each shot.

Meng: And each character gets an ID tag in those descriptions, so the annotation can track actions and appearances consistently across shots.

Jane: What I find impressive is the multi-stage verification. They don't just trust one model's output. They bring in Qwen3-VL-32B to check the reference images, discarding anything blurry or with multiple people.

Tom: There's also a leakage detection step where Qwen3-Max generates chain-of-thought rationales and then checks whether those rationales accidentally reference the original annotations.

Lu: That's such a crucial detail, because if the model's reasoning path copies the ground truth annotations, then you're not actually testing reasoning, you're testing memorization.

Meng: And then there's the question generation itself. They designed QA templates based on the six cognitive levels we mentioned earlier, and Qwen3-Max generates the actual questions conditioned on the annotations.

Tom: The outfit-change strategy is probably my favorite part of this page. They have two complementary approaches, one uses Qwen-Image-Edit to modify visual attributes while preserving the background, and the other uses face ReID to find different appearances of the same character.

Jane: That second one is elegant, because it's using real appearance variations that actually occur in the source material, not synthetic edits that might introduce artifacts.

Lu: And they cross-check identity consistency using both a visual model and a text-based model. The visual model describes the images, the text model compares those descriptions, and only samples that pass both checks survive.

Tom: It's a genuinely hybrid pipeline. Traditional vision tools for shot detection and face ReID, generative models for creating appearance variants, and LLMs for generating and verifying the annotations.

Meng: The paper calls the whole thing end-to-end automated, but the phrase "semi-automatic" appears too, and honestly that's the accurate description, because there are human checks embedded throughout.

Jane: They mark those checks as "check X" in their pipeline diagram, and they're placed after each critical step, which is how they get the quality level needed for both training and evaluation.

Lu: One thing I wonder about is the source material itself. They mention films, TV series, and anime, and those are all copyrighted content, but that's a discussion for another day.

Tom: Sure, let's keep the focus on the method. By the end of page three, they've established the training data pipeline, and the benchmark construction is actually much simpler because the training data was already vetted.

Jane: And that's a smart design choice. Because ISYV-75K went through all those automated checks, the benchmark only needs manual validation by a professional annotation team.

Meng: Which brings us to page four, where they explain the benchmark itself, the manual annotation protocol, and how it compares to the rest of the field.

Page 4 of the paper — Discuss page 4 of the paper: what is new on this page, explained in simple terms. Do not repeat the thesis already covered; build on it.: Meng: Page four starts with the benchmark construction, and the key word there is "manual." They use Qwen3-VL-32B to filter out overly simple samples, but then a professional annotation team takes over.

Tom: And the quality control is strict. Each sample is cross-validated, error-analyzed, and revised by at least three annotators, followed by a second-round spot check after the full review.

Jane: That's the kind of rigor you rarely see in automatically generated benchmarks. Most papers just trust the model and move on.

Lu: The comparison table on this page is where the benchmark's unique position becomes clear. They list eleven mainstream benchmarks, from TVQA to CrossVid, and ISYV is the only one with the multi-input and cognitive hierarchy columns checked.

Meng: And duration matters too. ISYV clips average around 50 seconds, which is shorter than Video-MME's 1,017 seconds but longer than most of the others.

Tom: But the paper argues it's not just about duration, it's about what happens within those 50 seconds. Complex shot transitions, costume changes, and cross-scene tracking.

Jane: There's a specific example they emphasize, Level 2 tasks where a character appears in a different outfit or location, and that's where models struggle the most because it tests what psychologists call object permanence.

Lu: Object permanence is such a great framing. It's the cognitive skill where you understand that an object continues to exist even when you can't see it, and models apparently have a hard time with that for people.

Tom: They also list the six levels and their cognitive analogs, basic perception, object permanence, procedural observation, social cognition, spatial memory, and causal reasoning.

Meng: That progression is actually pretty beautiful. You start with simple identification, move through tracking and observation, and end up at understanding why someone did something.

Jane: And it maps directly to the benchmark's design because they want to evaluate models the way you'd evaluate a human's developing understanding of other people.

Lu: The table also shows the annotation type, M for machine and A for human, and ISYV is the only one with both M and A. That hybrid approach is their answer to the scalability problem.

Tom: The manual verification is what makes the benchmark trustworthy, but the automated pipeline is what makes it possible to build 75,000 training samples without bankrupting the project.

Meng: And they're explicit about that trade-off. The benchmark samples were filtered for being challenging, not just any random sample, so it's deliberately hard.

Jane: Then, at the end of page four, they make the claim that the benchmark extends cognitive science-oriented evaluation, and I think that's the contribution that will age the best.

Tom: With the benchmark defined, page five pivots to the actual model. They introduce the ICQ Module and the training strategy, and this is where the engineering gets interesting.

Page 5 of the paper — Discuss page 5 of the paper: what is new on this page, explained in simple terms. Do not repeat the thesis already covered; build on it.: Tom: Page five introduces their solution to a very specific technical problem. When you give a model a video and a reference image, existing models often treat the reference image as the last frame of the video.

Jane: And that conflation wreaks havoc on the model's understanding. It literally cannot tell where the reference ends and the video begins.

Lu: Their fix is the ICQ Module, a small set of learnable tokens that go through self-attention and then cross-attention with the image features. The result is a compressed representation of the reference person.

Meng: And compression is the key word, because the raw image encoder produces a lot of tokens, and most of those tokens are background texture and lighting information that's just noise for this task.

Tom: By squeezing the reference image down through learnable queries, they keep the person-identifying features while dropping the irrelevant visual clutter.

Jane: Which also has a computational benefit. Fewer tokens means less processing, so the model can be trained and run more efficiently.

Lu: The training strategy follows a two-stage approach. First, supervised fine-tuning as a cold start, where the model learns the output format and basic knowledge. Then, reinforcement fine-tuning with GRPO to push performance higher.

Meng: And how they split the data between those stages is thoughtful. They use Qwen3-32B to score each chain-of-thought sample on a scale from 0 to 10 based on reasoning quality and leakage risk.

Tom: Samples with high leakage risk go into the RFT set, not the SFT set, because if the CoT explicitly references the original annotations, then supervised training would directly teach the model to copy those annotations.

Jane: It's a contamination-aware data split, and it's exactly the kind of detail that separates a rigorous dataset from a sloppy one.

Lu: The reward function then gets decomposed into four parts. There's the format reward that checks whether the model outputs the caption, candidate, think, and answer tags in the right order.

Meng: That weighted scheme gives the answer tag the highest weight at 0 point 5, which makes sense, since ultimately you want the right answer, but you also want the structure to hold up.

Tom: Then there's the accuracy reward, which is simply whether the final answer matches ground truth, and the caption semantic reward, where a judge model scores whether the caption conflicts with the reference description.

Jane: The clever part is that caption reward, because it prevents the model from ignoring the reference image entirely. If the model never describes the image, it gets no reward.

Lu: And then there's the ESR reward, the effective shot re-reasoning. This is the one that makes the model learn which shots matter without ever being told the ground-truth shot annotations.

Tom: The mechanism is to take the shots the model itself calls out in its candidate field, clip and concatenate those shots, then re-run the reasoning. If the second answer matches the first and both are correct, the model gets a bonus.

Meng: So effectively, the model is rewarded for finding evidence that supports the correct answer, and discouraged from padding its candidate list with irrelevant shots.

Jane: That's genuinely novel as far as I know. It's a self-supervised signal for evidence selection, and it requires no additional human annotation effort.

Lu: The mathematical definition on this page also includes this extra reward that decreases linearly with the number of output shots, which directly incentivizes fewer, more targeted shot selections.

Tom: I want to flag something in that math, though. When the reference person appears in every shot, the ESR reward degenerates to a binary score. So it only really kicks in for videos where the person disappears at some point.

Meng: And that's actually the right condition, because those are exactly the videos where evidence selection matters most. If you can see the person the whole time, you don't need to find the right shots.

Jane: Right. Page five gives us the architecture and the reward design. Page six is where they start running the actual experiments, and the setup is meticulous.

Page 6 of the paper — Discuss page 6 of the paper: what is new on this page, explained in simple terms. Do not repeat the thesis already covered; build on it.: Jane: So page six opens the experimental setup, and the first thing they establish is the evaluation protocol. All models get 32 frames uniformly sampled across the entire video duration.

Tom: That standardization is important because it means every model is seeing the same temporal coverage, so comparisons are actually fair.

Lu: And they're testing a wide range of models. Closed-source systems like GPT-5 point 2-global and Gemini-2 point 5-Pro, plus a spectrum of open-source models from Qwen2 point 5-VL-7B up to VideoLLaMA3-7B and InternVL3-8B.

Meng: The input format is also standardized. Every model has to produce a caption of the reference image, then a candidate shot list, then a thinking process, and finally an answer.

Tom: But not all models support thinking mode, so those only need to output the answer component. That's a sensible accommodation for the architectural differences.

Jane: They also introduce a second metric here, ICQ QA accuracy, which only counts answers where the model actually demonstrates engagement with the reference image.

Lu: And that matters more than you might think, because the paper found some models completely ignore the image and just hack the question using the video alone.

Meng: One example on this page shows Qwen2 point 5-VL-32B doing exactly that. It gives a plausible answer, but its image description is completely wrong, so it's basically guessing.

Tom: The baselines also include related work like IDA-VLM and PLVM, which are image-level personalization models adapted to this video setting, and Video-R1, which is a video reasoning RL system.

Jane: And they mark which baselines are evaluated under their original training setup versus which ones use the ISYV training setup. That distinction is crucial for interpreting the results.

Lu: The table on page seven is where the numbers come out, and there's a lot to unpack there. But first, let's look at the human performance baseline.

Meng: Human accuracy is 95 point 13 percent overall, and it's above 90 percent on every single level. The lowest is spatial memory at 90 point 81 percent, but even that is far above any model.

Tom: That human benchmark also involved blind evaluation by three annotators, marked with a star in the table, which gives us confidence the ground truth is solid.

Jane: So the setup is rigorous, and the result is that all models fall dramatically short of humans. Gemini-2 point 5-Pro is the best closed-source model at 67 point 10 percent, but that's still a 28-point gap.

Lu: The open-source models tell an even starker story. Most land between 20 percent and 40 percent, with Qwen2 point 5-VL-32B leading the pack at 39 point 29 percent.

Meng: And that's a 32B model. When you drop to 7B, you're looking at 24 point 76 percent for Qwen2 point 5-VL-7B with the image input format.

Tom: VideoLLaMA3's performance is almost comical in context. It gets 37 point 25 percent on the overall accuracy, which is respectable, but its ICQ QA accuracy collapses to 3 point 34 percent.

Jane: That means the model is answering correctly without ever properly describing the reference image. It's a perfect example of answer hacking.

Lu: The trained models at the bottom of the table tell the hopeful story, though. After RFT, their model hits 57 point 01 percent, which is actually slightly ahead of some closed-source performance on the overall metric.

Tom: But the level-by-level breakdown reveals where those gains come from, and that's what we need to dig into next.

Page 7 of the paper — Discuss page 7 of the paper: what is new on this page, explained in simple terms. Do not repeat the thesis already covered; build on it.: Tom: Page seven has the full results table, and this is where the level-by-level analysis really shines. Let's start with Level 1, basic perception, where humans are at 96 point 3 percent.

Jane: Gemini-2 point 5-Pro gets 85 point 93 percent, which is decent, but the open-source models struggle. Qwen2 point 5-VL-32B with image input gets 57 point 78 percent, and the 7B models mostly sit below 40 percent.

Lu: Level 2 is the object permanence tasks where the person changes outfit or appears in a new location, and this is where things get brutal.

Meng: Humans score 95 point 02 percent, Gemini gets 65 point 90 percent, and the best open-source model drops to 36 point 78 percent. That's a massive gap.

Tom: The paper interprets this as a fundamental limitation in cross-domain identity matching, which is one of the three core challenges they defined on page one.

Jane: Level 3, subtle details, is interesting because humans score 96 point 88 percent, but even Gemini drops to 66 point 25 percent, and most open-source models are below 30 percent.

Lu: Level 4, social cognition, involves gaze, dialogue, and social dynamics, and the pattern continues with humans at 96 point 11 percent and Gemini at 80 point 56 percent, but open-source models hovering around 20-30 percent.

Meng: Level 5, spatial memory and movement paths within and across shots, is where even Gemini collapses to 40 point 99 percent. Humans are at 90 point 81 percent.

Tom: And Level 6, causal reasoning, is another human-dominated category at 96 point 97 percent, with Gemini at 82 point 32 percent. Interestingly, Qwen2 point 5-VL-32B gets 58 point 59 percent here, which is relatively strong.

Jane: The ISYV-Model-RFT row is remarkable across all these levels. It hits 62 point 22 percent on Level 1, 61 point 69 percent on Level 2, 60 percent on Level 3, and 70 point 20 percent on Level 6.

Lu: But Level 4 remains its weakness at 37 point 22 percent. So social cognition, understanding gaze and dialogue dynamics, that's where the RL training didn't help much.

Meng: And Level 5 spatial memory is 50 point 18 percent, which is a big improvement over the base but still a clear weakness.

Tom: The comparison between Qwen2 point 5-VL-7B-SFT and ISYV-Model-SFT shows the architectural contribution of the ICQ Module. With the same SFT data, the ICQ Module improves accuracy from 28 point 03 percent to 33 point 55 percent.

Jane: And then the RFT stage takes it from 33 point 55 percent to 57 point 01 percent. So the RL training is responsible for the largest single gain.

Lu: The ablation table on page eight confirms this progression. Without any candidate learning in SFT, accuracy actually increases slightly in SFT-only evaluation, but then the RFT ceiling drops.

Meng: That's the essential trade-off. Teaching candidate output in SFT hurts immediate SFT performance but enables the ESR reward during RFT, which pays off massively in the end.

Tom: Adding the caption reward moves from 52 point 06 percent to 55 point 44 percent, and then adding the ESR reward pushes it to 57 point 01 percent. So each component contributes.

Jane: And the token length ablation shows 32 learnable tokens is the sweet spot. Fewer tokens lose information, more tokens introduce noise.

Lu: That result is a nice validation of the compression idea. You need enough capacity to represent the person, but not so much that you're carrying background clutter.

Meng: So page seven gives us the headline numbers, and page eight has the ablations and a case study that really illustrates what's going wrong in the baselines.

Page 8 of the paper — Discuss page 8 of the paper: what is new on this page, explained in simple terms. Do not repeat the thesis already covered; build on it.: Meng: Page eight continues with the frame count ablation, and the result is counterintuitive at first. Increasing from 32 frames to 64 frames doesn't help, it actually drops slightly to 56 point 56 percent.

Tom: The paper explains that this is because training used 32 frames, so the model learned to work with that temporal sampling density.

Jane: But there's a deeper lesson there too. When the model already captures the relevant events at 32 frames, adding more frames just adds redundant information.

Lu: Reducing to 16 frames drops performance significantly to 34 point 50 percent, which confirms that the model genuinely needs enough temporal coverage, 32 is the right balance.

Meng: Then we get to the case study, which is a Level 5 spatial memory question. And the comparison across models is honestly a little painful to read.

Tom: GLM4 point 1V-9B and Qwen2 point 5-VL-32B both misinterpret the reference image. Qwen3-VL-8B claims the referenced person doesn't appear in the video at all.

Jane: And InternVL3-8B fails to follow the required output format entirely. It's a perfect illustration of how different models fail in different ways on this task.

Lu: The most interesting case is Qwen2 point 5-VL-32B. It selects the correct option, but its image description is inaccurate. The paper argues this is a form of hacking, because the model didn't actually understand who it was supposed to be tracking.

Tom: And that's why the ICQ QA metric matters. If you only looked at answer accuracy, you'd think Qwen2 point 5-VL-32B was performing well on this sample, but the deeper analysis reveals the shortcut.

Meng: In contrast, the ISYV-Model both describes the reference person accurately and answers correctly, which is the behavior the whole training pipeline is designed to encourage.

Jane: The frame count results and the model comparison together tell us something important about where the bottlenecks actually are. It's not just about seeing more frames or bigger models.

Lu: Right, the bottleneck is the alignment between the reference identity and the video content. That's where the ICQ Module and the caption reward are specifically targeted.

Meng: And the ESR reward for shot selection addresses the second bottleneck, knowing which parts of the video to focus on.

Tom: So page eight completes the empirical story. The ablations show each piece of the framework contributes, and the case study shows how those pieces manifest in practice.

Jane: Now, page nine is the conclusion, and it honestly feels a bit short compared to the depth of the rest of the paper, but it does set up the future directions.

Lu: Let's see what they think comes next.

Conclusion — Tom and Jane summarize the paper 'I Seek You in Videos: Identity-Conditioned Queries for Person-Centric Video Reasoning' and its implications, say goodbye to the paper and get ready to discuss the next one. Do not introduce new facts.: Tom: So we've made it to the end of the paper, and the conclusion does what it should, it pulls the whole package together.

Jane: Let me try to summarize without repeating too much. The paper defines a new task, builds a benchmark with 1,377 samples, creates a 75,000 sample training set, and proposes a model with a specialized compression module and a four-part reward function.

Lu: And the key empirical result is that existing models, even the biggest proprietary ones, are far from human performance. The gap is largest in cross-domain identity matching and long-horizon tracking.

Meng: Their own 7B model reaches 57 percent overall accuracy after reinforcement fine-tuning, which is competitive with closed-source models in some aspects, though not all.

Tom: What do you think the real-world impact is going to be, Jane?

Jane: I think the ICQ task definition alone is going to get adopted by the community. It's a natural extension of video reasoning, and now that there's a benchmark, people will start building against it.

Lu: The ESR reward is probably the most transferable idea. The concept of rewarding a model for re-reasoning over the evidence it selected, without needing ground-truth evidence annotations, that could apply to many domains beyond person tracking.

Meng: I'd also point to the six-level cognitive hierarchy as a framework that other benchmark builders will borrow. It's a principled way to organize difficulty and interpret model failures.

Tom: The paper also hints at future work, expanding to more complex and realistic scenarios. I suspect we'll see a follow-up that extends this to egocentric video or multi-person simultaneous tracking.

Jane: And I'd love to see the dataset construction pipeline, the multi-model verification, the leakage detection, that pipeline is a blueprint that other groups can adapt for their own domains.

Lu: One thing I'll be watching is whether the open-source community can reproduce the 57 percent result with different base models. The framework should generalize, but that's worth testing.

Meng: The paper leaves us with the acknowledgment that we're still a long way from human performance in this kind of reasoning, and that's genuinely motivating. There's a real gap to close.

Tom: Alright, I think we gave this one a fair hearing. It's a solid package with real implications, and I'm curious to see who picks it up first.

Jane: Agreed. And with that, let's say goodbye to this paper and get ready to see what's next up for discussion.

Tom: Thanks for joining us, everyone. Onward to the next paper.

More episodes

← Home