Track and Caption Any Motion: Open-Vocabulary Spatiotemporal Captioning via Trajectory-Conditioned Generation

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Track and Caption Any Motion".

Jane: Here's the gist of the paper. The gist The model presents TCAM, a generative framework that watches a video and with no text query and no region prompt decides what is moving, describes each motion in open vocabulary, locates it in time, and points to the exact trajectories that carry it.

Tom: How it works TCAM couples tracking and language at point granularity through a Caption-Aware Resampler (CAR), where a small set of learnable queries crossattends to dense point trajectory tokens and distills them into a fixed-length motion context that conditions a language decoder.

Jane: The decoder generates an entire video’s events in a single pass, each with a free-form caption, a start and end time, and a pointer to the trajectories it refers to, for sequential events and several subjects active at once TCAM couples tracking and language at point granularity through a Caption-Aware Resampler (CAR), where a small set of learnable queries crossattends to dense point trajectory tokens and distills them into a fixed-length motion context that conditions a language decoder The architecture consists of three sequential modules: a Trajectory Tokenizer, a Caption-Aware Resampler (CAR), and a Structured Generator The Trajectory Tokenization step extracts dense point trajectories using frozen CoTracker3 initialized on a 36 × 36 regular grid (K = 1296 tracks), encoding position, velocity, and visibility into track tokens These tokens are then enriched by cross-attention against frozen ViT-B/16 frame features to produce appearance-aware motion tokens called VisionMotionEncoder The Caption-Aware Resampler (CAR) compresses the large trajectory tokens into a fixed-length context through M learnable query vectors Q(0) that are refined over L = 4 stacked layers, where each layer involves a cross-attention against VisionMotionEncoder output followed by a feed-forward sublayer The output of this process yields the motion context vectors Z, which are then prepended to per-frame temporal tokens to restore frame identity Key Contributions and Training TCAM achieves trajectory-conditioned generation by conditioning an unmodified FLAN-T5 decoder on dense point-tracking tokens through M learnable queries, enabling open-vocabulary description of motion at point granularity, without a fixed text bank, a user query, or a region prompt It also features grounding on a shared backbone where the generated token fires a lightweight pointer head that scores the same dense trajectory tokens that condition the caption, so grounding is not a separate detection stack The training uses a three-term objective: L = Lcap + λgrd Lgrd + λdiv Ldiv, which supervises caption quality (Lcap), spatial pointer alignment against object masks (Lgrd), and trajectory diversity (Ldiv) The grounding loss is supervised using existing segmentation annotations, requiring no manual event labeling Performance and Evaluation TCAM was evaluated on three axes corresponding to the three capabilities of the system: (i) spatiotemporal grounding on MeViS Val and Ref-YouTube-VOS Val (J, F, J&F); (ii) dense video captioning on ActivityNet Captions (SODA c, CIDEr, METEOR); and (iii) trajectory pointer accuracy on TAP-Vid DAVIS and TAPVid Kinetics (AJ, OA, δavg) TCAM reaches 61.1 J&F on Ref-YouTube-VOS Val while using a smaller ViT-B/16 backbone compared to methods like ReferFormer Furthermore, TCAM improves over the strongest baseline, Streaming Vid2Seq, on all three metrics (6.5 vs.

Tom: 6.2 SODA c, 39.4 vs.

Paper summary: Tom: Welcome back to the show. Today we’re looking at TCAM which stands for Track and Caption Any Motion: Open-Vocabulary Spatiotemporal Captioning via Trajectory-Conditioned Generation. Jane, what’s the core idea here in a nutshell?

Jane: So basically, the thesis of this paper is that they want a model that can watch a video without needing any text query or region prompt and still produce descriptions of what’s moving in open vocabulary, plus tell you exactly where those motions are happening over time. It claims to combine tracking and language at the point level for motion granularity.

Tom: So it’s this idea of combining those two things, the physical tracking part and the language generation part, in one unified system. What does that mean for us right now?

Lu: The core mechanism they propose is a Caption-Aware Resampler, or CAR. This component uses a small set of learnable queries to crossattend to dense point trajectory tokens and distills them into a fixed-length motion context that conditions the language decoder. If this works as described, it means you can get rich descriptive output just by watching raw video without any prior text guidance.

Meng: From an engineering standpoint, conditioning a language decoder on a compressed context from dense trajectories sounds heavy on the representation learning side. What are they doing to make sure that compression doesn't lose too much detail when you’re trying to capture sequential events?

Lalam: Lalam here, and I can tell you what it says: they use the Caption-Aware Resampler to compress those dense trajectory tokens into a fixed-length context. This context then conditions an unmodified FLAN-T5 decoder, allowing for open-vocabulary description of motion at point granularity without needing a fixed text bank or a user query.

Jane: So, the naive question someone might ask is how they actually manage that transition from raw video data to those structured tokens that feed the language model. How do you even get those dense point trajectories represented?

Tom: Well, they use CoTracker3 initialized on a thirty-six by thirty-six regular grid with K equals one thousand two hundred ninety-six tracks, encoding position, velocity, and visibility into track tokens. They then enrich these tokens by cross-attention against frozen ViT B/sixteen frame features to produce what they call appearance-aware motion tokens.

Lu: That ViT B/sixteen part is important because it’s how they get the appearance awareness into the motion tokens, which feeds into that VisionMotionEncoder. It’s about making sure the motion understanding isn't just purely positional.

Meng: If you’re deploying this, what does running this actually cost? You mentioned training on existing captions and segmentation annotations without additional event labeling. How robust is that training regime when you try to scale it up to real video traffic?

Paper summary: Lalam: They use a three-term objective: L cap plus lambda grd plus lambda div, which supervises caption fidelity, spatial pointer alignment against object masks, and trajectory diversity using only standard segmentation annotations. They also mention scaling to PLM Video Human improves grounding precision as the training set grows.

Tom: That’s interesting because they aren't doing any manual event labeling. The grounding loss is supervised using existing segmentation annotations, which means they are leveraging existing data structures, right?

Jane: Exactly, and that’s a big deal for accessibility. It means you can train this on whatever video data you already have labeled for segmentation without needing to hire people just to label every single motion event. The results show it outperforms dense captioning baselines on ActivityNet Captions and approaches query-based grounding methods on Ref-YouTube-VOS, all without training on those specific datasets.

Lu: The performance metrics are also interesting. They report sixty-one point one J andF on Ref-YouTube-VOS Val while using a smaller ViT B/sixteen backbone compared to some other methods like ReferFormer, which suggests efficiency in the representation learning part.

Meng: Efficiency is key for deployment. What about the pointers? When they say pointer selection exceeds chance by five point three to eight point five times, does that mean we actually get useful visual evidence back from the system, or is it just random noise that happens to look good in a diagnostic?

Lalam: The diagnostics show that the pointer selection exceeds chance by five point three to eight point five times, and the pointer mass concentrates on moving sub-parts above a mask uniform reference. These selection diagnostics are distinct from full query tracking accuracy.

Tom: That distinction is important because it tells us *how* the model is grounding things, not just *if* it found something. So if you only listen to this, what’s the big picture change here for someone who isn't deep in the mechanics?

Jane: Well, what this paper presents is a system that can see a video and give you descriptions of everything happening in it—sequential events, concurrent subjects—and it doesn't ask you for any prior text prompt or region selection. It just watches and describes the motion directly.

Lu: This work sits at the intersection of point tracking, videolanguage models, dense video captioning, and generative multimodal reasoning. It’s trying to unify spatial trajectory precision with open vocabulary generation in a single query-free pass.

Meng: From what I'm hearing about the architecture, the component ablation studies show that removing the Caption-Aware Resampler degrades caption quality the most of any row, suggesting that this compression step is absolutely critical for maintaining high description fidelity.

Lalam: And when scaling to PLM Video Human, they confirmed that this architecture absorbs more data without needing task-specific engineering, showing consistent gains on both spatiotemporal grounding and captioning quality as training data grows.

Tom: So, it seems the implication is that we might be able to build video understanding systems that are much more self-contained and descriptive than what we see in current models that rely on explicit prompts.

Paper summary: Jane: That’s right. The authors call TCAM the first model to deliver point-level grounding, temporal structure, and open vocabulary generation in a single query-free pass: tracking and language, together, at motion granularity. It shows how you can combine these capabilities without relying on a fixed text bank or any external user input.

Lu: It’s about showing that you can achieve this level of detail by conditioning the language model on dense point tracking tokens through M learnable queries, which enables open-vocabulary description of motion at point granularity.

Meng: If we think about deployment, the fact that they are evaluating it on benchmarks native to each axis—TAP-Vid for space and time, MeViS / Ref-YouTube-VOS for spatial localization, and ActivityNet Captions for dense captioning—suggests a holistic evaluation framework.

Lalam: And they also show that pointer selection exceeds chance by five point three to eight point five times, which is a strong signal when you're looking at the grounding aspect of this model.

Tom: So, we’ve covered how TCAM uses its Caption-Aware Resampler to bridge the gap between dense trajectories and language decoding, and how it handles training with three different loss terms. It’s a lot of moving parts packed into one framework.

Jane: And ultimately, what this means for us is that video understanding could move towards a system where describing complex motion doesn't require you to pre-define what you’re looking for, but the model figures out the story and the physical paths on its own.

Lu: The paper really highlights how different prior works occupy the two-axis corners: point trackers localize in space and time but emit no language; referring segmentation grounds in space from a language query; dense captioners describe events in time and language from clip-level features. TCAM reaches the remaining corner, combining point-level grounding, temporal structure, and open vocabulary generation at once.

Meng: I just wonder what the practical limits are when you move this from these high-quality benchmarks to videos that are noisy or have very long temporal dependencies where tracking might drift significantly.

Lalam: The paper does acknowledge limitations: they rely on existing segmentation annotations for the grounding loss, and scaling to PLM Video Human improves grounding precision as the training set grows, which suggests it’s still heavily dependent on having good labeled data for those specific tasks.

Tom: So to wrap up, TCAM is this unified approach that uses trajectory tokens and a learnable query mechanism to condition language generation on motion data directly. It’s a significant step toward truly open-vocabulary video understanding without explicit prompting.

Jane: And the conclusion is that it delivers point-level grounding, temporal structure, and open vocabulary generation all in one query-free pass: tracking and language together at motion granularity. It really shows how you can combine these different modalities effectively when the goal is to describe complex motion directly from raw video.

Conclusion: Tom: So we’ve been looking at TCAM, which stands for Track and Caption Any Motion: Open-Vocabulary Spatiotemporal Captioning via Trajectory-Conditioned Generation.

Jane: Right, so the main thing is this model can watch a video without any text prompt or region selection and produce captions about what’s moving, and it also tells you the exact paths those motions are taking over time.

Lu: The core idea is using a Caption-Aware Resampler to take dense point trajectory tokens—which track every single point in motion—and distill them into a compact context that conditions a language decoder.

Meng: And that’s what lets it generate the whole sequence of events in one pass, each with its own caption and its own pointer to the path.

Lalam: They train this using existing captions and segmentation labels, which is cool because they don't need anyone to manually label every event.

Tom: The training objective involves supervising caption quality, spatial pointer alignment against object masks, and trajectory diversity all at once.

Jane: So the results show it achieves sixty-one point J andF on one benchmark while using a smaller vision backbone compared to some other methods that might be bigger.

Lu: And component ablation studies show that if you take out the Caption-Aware Resampler, caption quality drops significantly, which tells us that compression is really critical for keeping the detail.

Meng: If we were trying to deploy this right now, I’d worry about how it handles noisy video or very long sequences where tracking might drift.

Lalam: The authors say they rely on existing segmentation annotations for the grounding loss, so it's still tied to having good labeled data for those specific tasks.

Tom: The paper concludes that TCAM is the first model to do point-level grounding, temporal structure, and open vocabulary generation all in one query-free pass: tracking and language together at motion granularity.

Jane: It really moves us toward a system where describing complex motion doesn't require you to pre-define what you're looking for, but the model figures out the story and the physical paths on its own.

Lu: This work is about showing that you can combine point tracking precision with language generation through learnable queries to achieve open vocabulary description at a very fine level.