2608.09435-Listen, See and Track: Spatio-Temporal Audio-Visual Sound Event Reasoning for Omni-Modal Language Models

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Listen, See and Track: Spatio-Temporal Audio-Visual Sound Event Reasoning for Omni-Modal Language Models".

Jane: The paper was written by Zhi Zeng, Cheng Zhang, Zesheng Yang, Rendong Pi, Jiaying Wu et al. from Xi'an Jiaotong University and The Hong Kong Polytechnic University and National University of Singapore and Central China Normal University and University of California San Diego.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper summary: Tom: Welcome back. Today we've got a paper that tackles a question most of us don't even think about, but our brains solve constantly, which sound goes with which thing you can see moving around you.

Jane: Right, and that's exactly what they call tracking and binding. The paper introduces a new benchmark called ST-OmniQA and a model called ST-Omni-R1. The goal is to make language models understand not just what sound is happening, but where the source is, how it moves, and which visible object it belongs to.

Tom: And why is that hard? Because existing audio-language models treat a clip as one global sound event. They hear the whole thing and say, footsteps, but they can't tell you if the footsteps are approaching from the left or moving toward the person in the doorway.

Jane: Exactly. Meanwhile vision-language models have no spatial audio cues at all. So the paper builds a dataset of 40,000 panoramic videos with spatial audio, and 400,000 question-answer pairs. And the questions are organized into four difficulty levels, from recognizing a single sound source all the way up to tracking a source that disappears behind an occluder and reappears.

Tom: Let's bring in Lu. You've been quiet.

Lu: I was just processing the scale, Tom. 400,000 questions. That's not a toy. And they generated it with full simulator control, so they know the ground truth for every sound source at every moment. That's what lets them train the model with what they call executable reasoning graphs.

Meng: And the model itself, ST-Omni-R1, starts from a spatial audio encoder and fuses its output with video from a Qwen2 point 5-VL backbone. Then they train it progressively, stage by stage, and then use reinforcement learning to sharpen the reasoning. The result is 77 point 83 percent average accuracy, versus 37 point 28 percent for the best baseline they compared against.

Tom: That's a massive jump. But hold on, let's be careful. The baselines aren't trained on this dataset, right?

Lu: Right, they're evaluated zero-shot. So part of the gap is the advantage of training on the benchmark. Still, the more interesting result is that their spatial and motion representations transfer to three real-world spatial audio benchmarks. That suggests the model isn't just memorizing the simulator.

Jane: And that's the big picture. Lalam, you're the one who usually zooms out.

Lalam: The bigger picture is that this moves audio understanding from answering, what sound is this, to answering, which object is making this sound, where is it moving, and what happens when I can't see it. That's the kind of capability you'd need for a robot navigating a room, or for hearing aids that help people track a conversation partner in a crowded café.

Tom: So there's a real application hook there. But how did they actually construct the benchmark so that the questions force genuine audio-visual reasoning? That's the part I want to dig into in the next segment.

Jane: Good, because page one sets up exactly that problem.

Page 1 of the paper: Tom: So we've said the paper is about making models track sound sources, but page one really lays out why this is a genuine research problem and why it's been missing.

Jane: And it does it with a really concrete example. Imagine a person walking toward a room where someone else is seated. Before the walker appears in view, you can hear the footsteps getting louder and changing direction. That's the auditory looming effect. But once the walker enters the doorway, your brain binds the sound to the visible person and tells them apart from the stationary occupant.

Lu: And the key claim is that the event label footsteps plus isolated visual frames can't establish that correspondence. You need joint reasoning over what the sound is, where it comes from, how it's moving, and what visible thing matches it. The paper calls this a tracking and binding problem.

Meng: They also identify three specific gaps. The perception gap is recovering time-varying source geometry, the binding gap is associating acoustic trajectories with visible identities, and the reasoning gap is composing spatial, temporal, and cross-modal evidence. I like how they separate those, because they really are different skills.

Tom: And that phrasing, perception, binding, reasoning, it's a nice ladder. But what's the evidence that existing models are bad at this? Page one mentions spatial audio models like BAT, OWL, SPUR, Spatial-Omni, and then SELD systems. They all do pieces, but none do the whole thing.

Jane: Right. BAT and OWL handle direction and distance, but mostly for static scenes. SELD systems like SALSA track moving sources, but they output fixed event labels and frame-level locations, so they can't do open-ended reasoning about landmark relations or occlusion. And the recent audio-only models, like ST-AudioLM, still don't bind trajectories to visual instances.

Lalam: And that's the missing piece. The human brain does this by integrating what you hear with what you see, and the correspondence is built across time, not just at one moment. So the paper's claim is that no current omni-modal model has been explicitly trained to do this, and there wasn't even a benchmark to measure it.

Tom: So the motivation is clear. But now I have to ask, how do you even build a benchmark that forces a model to use both senses together? Because if you write questions, you have to make sure the audio alone can't give away the answer.

Jane: Exactly, and that's the "modality necessity constraint" they mention in the abstract. We'll see the details on page three, but the idea is they only keep questions where the answer is ambiguous if you look at audio alone or video alone. The answer becomes unique only when you combine them.

Meng: That's clever, because it prevents shortcuts. And it's connected to the way they construct scenes with same-class sources and visual distractors. But to understand how they actually render those scenes and generate the questions, we need to look at the benchmark section.

Lu: And before we get there, one thing that struck me on page one is the claim that humans resolve this by matching auditory and visual evidence. They cite work on multisensory integration from neuroscience. So the benchmark is conceptually grounded in how the brain actually works, not just in a heuristic about data.

Tom: That's a great observation, Lu. And it sets the stage for what's on page two, which is a survey of related work. Let's see how they position themselves there.

Jane: Yeah, let's move to page two.

Page 2 of the paper: Tom: So page two is basically the related work section, and it's where the paper positions itself against three research lines: sound event localization and detection, audio-visual source localization, and large audio-language models.

Jane: And the pattern is the same across all three. Each line does something useful, but each one stops short of the full problem. SELD systems like SALSA and PSELDNet track moving sources over time, but they produce fixed labels and coordinates, so they can't reason openly about what a source is doing relative to other objects.

Meng: Right. And audio-visual source localization, like the work from Owens and Efros and Senocak, learns to find which object makes a sound, but mostly for static images or dominant sources. They don't bind time-varying trajectories to multiple visible instances.

Lu: Then there are the large audio-language models, like Pengi, LTU, SALMONN, Qwen2-Audio, Kimi-Audio, Audio Flamingo. They are great at open-ended semantic reasoning, but they represent an entire clip as global acoustic content. They have no notion of direction, distance, or motion. Actually, wait, there is one exception in the survey: Tang et al. introduced multichannel spatial features into an LLM. But the paper says the predictions remain task-specific.

Tom: So essentially everyone does a piece. And then there's the spatial-audio language model line, which is the closest. BAT, OWL, SPUR, Spatial-Omni, those inject spatial cues into LALMs. But the paper says they emphasize static or clip-level spatial attributes. The question is who tried to model moving sources before.

Jane: That's where the two recent audio-only works come in. One is Spatial Audio Motion Understanding and Reasoning, from Sridhar, Guo, and Visser. The other is a concurrent paper called ST-AudioLM, from a Sony and KAIST group. ST-AudioLM learns time-resolved FOA representations with dense trajectory supervision. And the paper acknowledges it's especially related to their audio branch.

Lu: But here's the difference. ST-AudioLM is audio-only. It doesn't bind those trajectories to visible instances in a panoramic video. The paper says its setting requires acoustic trajectories to be bound to visible objects and composed with evidence about landmarks, occlusion, and other moving sources. That's the binding gap again.

Meng: And it's worth noting that with ST-AudioLM being concurrent, there's a race aspect here. Both are doing dynamic FOA representations, but this paper adds the visual binding dimension. That's a significant differentiator.

Tom: So they position ST-OmniQA as the first benchmark that tests all of it together. And they position ST-Omni-R1 to use FOA-derived trajectory tokens and panoramic visual context. But I want to make sure we understand what FOA actually is, because it comes up again and again on page three and four.

Jane: FOA stands for First-Order Ambisonics. It's a way of encoding spatial audio into four channels: one omni-directional and three that capture front-back, up-down, left-right intensity. You can derive direction and distance from those channels.

Lu: Exactly, it gives you a three dee sound field, not just stereo. That's what lets the model estimate azimuth, elevation, and distance over time.

Tom: Good. So page two is the setup, and it tells us what's missing. Now let's get into what the paper actually built. Page three describes the benchmark formulation and the data generation pipeline.

Jane: Let's go.

Page 3 of the paper: Tom: Page three is the heart of the benchmark design, and it's where the paper shows that ST-OmniQA isn't just a pile of videos with questions bolted on. There's a formal structure underneath.

Jane: Right, they define each sound source as a state vector that changes over time. For source i at time t, that state has the event identity, whether the source is active, its azimuth, elevation and distance, its motion state, its visibility, its visual object identity, and its relations to other sources and landmarks.

Lu: That's the S_i(t) equation in the paper. And the point of writing it down as a formal state is that every question can then be generated as a deterministic query over either one of those variables or a composition across sources and time and modalities. So the QA pairs are guaranteed to test something precise, not just vaguely about the scene.

Meng: And the scenes themselves are generated using Matterportthree dee meshes for indoor environments and SoundSpaces 2 point 0 for acoustic simulation. So you get realistic room geometry and realistic reverberation. Then they place one or more sound-emitting three dee objects in the scene and render a 10-second panoramic video with synchronized Ambisonics audio.

Tom: And they define five source configurations: single static, single dynamic, two static, one static and one dynamic, and two dynamic. That's what allows them to control difficulty across levels.

Jane: Exactly. And for moving sources, they render time-varying room responses along the trajectory. That's crucial, because it means the direction, distance, and reverberation are all synchronized with the visible motion. You can hear the source approach and see it approach at the same time.

Lu: Then they annotate 50 temporal states per clip. For each source, they have activity intervals, motion states, azimuth over time, elevation, distance. Plus visual annotations like bounding boxes and visibility states, and scene-level annotations like landmarks and occluders.

Meng: And then the four capability levels. Level A is single-source acoustic perception, event, activity, DoA, distance, motion state. Level B is multi-source spatial perception, which forces you to select the target source and disambiguate between competing sources. Level C introduces temporal and cross-source relations, trajectory comparisons. Level D is the hardest, binding, landmark grounding, occlusion, tracking after visual disappearance.

Tom: And the key thing is that Level D questions are filtered by that modality necessity constraint we mentioned. The candidate sets from audio alone and video alone both have more than one answer, but the joint evidence gives exactly one unique answer.

Jane: That's the |CA| > 1, |CV| > 1, |CAV| = 1 condition in the paper. It's a solid way to force genuine cross-modal reasoning. If audio alone could answer, the model could cheat by ignoring the video entirely.

Lu: And they also generate reasoning traces for Levels C and D, executable graphs that show which interval to select, how to bind the target, and how to compute the relation. Those traces become supervision for the reinforcement learning later.

Meng: One more detail I want to highlight: they split the data by scene-room unit to prevent leakage. So the same room doesn't appear in both train and test, which makes generalization meaningful.

Tom: That's an often-overlooked detail that makes the benchmark trustworthy. So page three gives us the data. Page four is where the model starts, with the FOA encoder and the spatial representation.

Jane: Let's get to page four.

Page 4 of the paper: Tom: Page four is where the model architecture comes in. And the first thing they do is define how to turn the raw Ambisonics waveform into something a neural network can chew on.

Jane: They use a four-channel FOA waveform in the AmbiX convention: W, X, Y, Z. And they reorder it to W, X, Y, Z in Cartesian order. Then they compute something called the normalized acoustic-intensity vector. That's a per-time-frequency estimate of where the sound energy is flowing.

Lu: And that's exactly what you need for direction. The intensity vector points toward the source. Then they combine four log-Mel spectrogram channels with Mel-projected intensity components, so the input feature is essentially both spectral content and directional cues at every time-frequency bin.

Meng: Then that feature goes into a channel-fusion layer and into an Audio Spectrogram Transformer. That's the AST architecture from Gong et al. The output is a set of time-frequency patches.

Tom: And here's where the paper gets creative. From those patches, they produce one global semantic token and a variable number of temporally ordered trajectory tokens. The semantic token summarizes the whole clip, what sound is happening. The trajectory tokens preserve how the source moves over time.

Jane: The way they get those trajectory tokens is by averaging the patch features over frequency, then interpolating the temporal sequence into K ordered bins, 40 in their case. Then temporal self-attention produces the trajectory tokens. Each token describes the source state in a particular time bin.

Lu: And the auxiliary supervision heads predict source activity, a unit direction vector, and log-distance for each bin. But those heads are only used during initialization, to teach the encoder what the tokens should mean. After that, they're removed.

Tom: So the encoder is trained to embed time-varying geometry into the token stream. And the key idea is that the language model receives one semantic token plus 40 trajectory tokens. So it has both the global event and the time-resolved motion.

Meng: And they align the geometric supervision with the benchmark states. Azimuth and elevation are converted to a unit direction vector, and distance is converted to log-distance. The trajectory loss combines binary cross-entropy for activity, an L2 loss on the direction vector, and an L2 loss on log-distance, masked by activity.

Lalam: And there's a clever trick to avoid catastrophic forgetting during that initialization. They keep the static encoder frozen as a teacher while doing the dynamic adaptation. So the model learns to preserve event semantics while adding localization. The objective includes a term that keeps the semantic token close to the pretrained one.

Tom: So the audio path is well-defined. But the full model isn't just audio. There's a video encoder and a connector that projects the audio tokens into the language model embedding space. That's what we'll see on page five, along with the curriculum training.

Jane: Right, page five is exactly where the training strategy starts.

Page 5 of the paper: Tom: Page five continues the model description, and it's where the paper explains how the audio and video paths come together in the language model.

Jane: They use a video encoder from Qwen2 point 5-VL-7B-Instruct to transform the panoramic video into visual tokens. Then for each audio token, a trainable connector projects it into the language model's embedding space. The projected audio tokens are inserted into the decoder context together with the visual tokens and the question tokens.

Lu: And the decoder can attend jointly to all of it. That's the fusion. The perceptual encoders are frozen during tuning, so only the connector and the language-side modules learn to align acoustic trajectories with visible objects and scene relations. That's a pretty standard recipe.

Meng: Then comes Stage I, the progressive curriculum. They organize training into four stages, Stage-A through Stage-D, matching the benchmark levels. Stage-A does single-source perception. Stage-B adds multi-source scenes and target selection. Stage-C introduces temporal and cross-source relations. Stage-D brings in the visual binding, landmark grounding, occlusion, and tracking.

Tom: And why not just train on everything at once? Because the paper argues that progressive curriculum gradually increases reasoning difficulty, so the model can build on each capability before tackling the next.

Jane: Exactly. And the loss is just standard next-token cross-entropy on the response tokens. Nothing fancy there. The tricky part is that the model needs to learn the right alignment between audio trajectory tokens and visual instances, and that's where the ordering of stages matters.

Lu: So the curriculum is a training strategy, but note the paper says these stages are distinct from the benchmark levels. The benchmark levels are evaluation capability groups, while the stages are a training process. That's an important distinction, because people might confuse Stage-D training with Level-D evaluation.

Meng: Right, they're aligned in name but not identical in function. And then after the curriculum, they run Stage II, which is reinforcement learning. That's the reasoning-tree part. The RL is designed to enforce consistency between intermediate reasoning steps and the final answer.

Tom: And that's where it gets really interesting, because they don't just reward the final answer. They build a tree of reasoning nodes that formalize the source states, trajectories, bindings, and relations. Then they score the sampled response as a path through that tree.

Jane: Let's hold that thought, because the full description of the tree reward and the group-relative policy optimization is on page six.

Tom: Perfect, then we naturally move to page six.

Page 6 of the paper: Tom: Page six is all about the reasoning-tree reinforcement learning, Stage II. And the setup is intricate, so let's break it down.

Jane: For each question, they build a task-specific tree from the benchmark annotations. The root encodes the multimodal context. Intermediate nodes formalize source states, spatial trajectories, audio-visual bindings, and scene relations. The terminal nodes specify candidate answers. So the tree is essentially a structured representation of how a perfect reasoner would solve the question.

Lu: And when the model samples a response, they interpret that response as a root-to-leaf path through the tree. Then they score it based on how well it matches the correct path.

Meng: The reward has three components. Format compliance, node-level reasoning consistency, and final-answer accuracy. The tree score is an average over the reasoning nodes of how correct they are, minus a penalty for violations of parent-child dependencies. So the model is rewarded for not just giving the right final answer, but following a reasonable chain of reasoning.

Tom: And that's different from typical RLHF where you only care about the answer. Here the intermediate steps matter directly.

Jane: Right. And then they use GRPO, Group Relative Policy Optimization, to update the policy. They sample eight responses per prompt, group them, compute each response's reward, normalize the reward within the group to get an advantage, and then do a clipped policy update similar to PPO but without a separate critic.

Lu: And the interesting part is the standard deviation in the denominator. If all eight responses in a group get the same reward, the advantage is zero, so the model doesn't update. The relative optimization signal only comes from groups that have within-group reward variation. That keeps the learning signal meaningful.

Meng: And because the model is sampling over the tree paths, it's effectively learning which reasoning paths are good. You can think of it as exploring the tree structure through sampling. And the format reward plus the semantic-equivalence evaluator for free-form responses allows valid linguistic variation.

Tom: So Stage II is about consistency. But the paper also includes ablation studies. Page seven is where the experiments are, and I want to see the actual numbers.

Jane: Yes, page seven is the results page, and there's a lot to unpack there. Let's move.

Page 7 of the paper: Tom: Page seven is the experiments section. And the headline result is in Table 2. ST-Omni-R1 with SFT plus reinforcement learning hits 77 point 83 percent average semantic accuracy, versus 37 point 28 percent for the best evaluated baseline.

Jane: And the best baseline is Gemini-3 point 1-Pro, a closed-source model. That's a huge gap. But I want to stress what we said earlier: those baselines are evaluated zero-shot, without task-specific tuning. So part of the gap is expected. Still, even the SFT-only version of their model scores 75 point 28 percent, which is still far above all baselines.

Lu: And the interesting thing is the breakdown by level. The baselines don't just do poorly overall, they collapse on Level B, multi-source spatial perception. Open-source models score below 30 percent there. The hardest part is distinguishing between competing sources in space.

Meng: Meanwhile ST-Omni-R1 is strongest at Level D, scene-grounded reasoning, at 94 point 7 percent. That's remarkable, because Level D is the hardest for baselines the best baseline gets 47 point 5 percent. The gap is enormous. And that's exactly where cross-modal binding matters most.

Tom: Then there's Table 3, the transfer evaluation on three real-world benchmarks. TAU-NIGENS and Lthree deeAS22 are audio-only, and they beat BAT on azimuth, elevation, distance, and motion. The biggest win is on STARSS23, which includes synchronized panoramic video, where they get 79 point 04 percent azimuth accuracy versus 49 point 70 percent for BAT.

Jane: And that's because STARSS23 has visual evidence to bind to, which BAT lacks. The paper is careful to say this larger margin reflects each model's supported modalities, not controlled audio-only superiority. So they're not claiming their audio is better than BAT in isolation, but that audio-visual binding is better.

Lu: Table 4 is the ablation. And the most striking row is the Stage-A-only model, which scores 4 point 00 percent on Level B. That confirms that without the curriculum stages, the model cannot handle multi-source scenes at all. Each stage adds measurable capability: 50 point 40, then 69 point 40, then 74 point 50, then 74 point 30 on Level B.

Meng: And the modality ablation shows video-only at 71 point 63 percent and audio-only at 47 point 00 percent. The full model at 77 point 83 percent beats both. So neither modality is sufficient, but combining them gives the best result. That's a clean demonstration of the paper's thesis.

Tom: So the experiments support the claims. But the last page is the conclusion, and it's short. There's not much new, but there's an important framing of the contribution.

Jane: Right, the conclusion says this advances audio-visual understanding toward source-level spatio-temporal reasoning. And the key sentence is about tracking sound sources as persistent multimodal entities whose semantics, geometry, motion, and visibility evolve over time. That's the conceptual shift.

Lu: And I'd add that the biggest implication is for embodied eye and robotics. If a robot can track a sound source as a persistent entity, even when it's occluded or when it's silent for a moment, then the robot can build a much more stable model of the world.

Lalam: And it also matters for human-computer interaction. Hearing aids, augmented reality glasses, smart assistants in a room. All of these could benefit from knowing not just what sound is happening, but which direction it's coming from and which object or person it's associated with.

Tom: I think the transfer results are the most encouraging sign for future work. The representations learned in simulation with 75 event classes transfer to real-world recordings. That suggests the approach is on solid ground.

Meng: That's right. And the community will likely build on this in a few obvious ways: expanding the event classes beyond 75, adding more complex multi-source interactions, or testing on egocentric videos where the listener moves.

Jane: And there's the question of whether the trajectory tokens could be made even finer-grained, or whether the semantic token could be dropped to reduce latency. The paper leaves a lot of room for optimization.

Tom: Well said. I think we've covered the benchmark, the model, the training, and the results. And we've highlighted why it matters for real-world systems.

Conclusion: Tom: Alright, let's wrap up. We've spent this episode on a paper that builds a benchmark and a model for spatio-temporal audio-visual reasoning. The main contribution is simple to state: it makes models track sound sources as persistent entities with a location, a trajectory, and a visual identity.

Jane: And they backed it up with a lot of engineering. 40,000 videos, 400,000 questions, four capability levels, a curriculum and a reinforcement-learning stage. The result, 77 point 83 percent average accuracy versus 37 point 28 percent for the best baseline, is a strong demonstration that the approach works.

Lu: I think the most elegant idea is the modality-necessity constraint. By filtering questions so that neither audio alone nor video alone gives a unique answer, they force genuine cross-modal reasoning. That's what makes the benchmark trustworthy.

Meng: And the transfer to real-world benchmarks suggests the model isn't just memorizing simulator quirks. Getting 79 percent azimuth accuracy on STARSS23 when BAT gets 49 point 7 percent is a meaningful result.

Lalam: The bigger message is that this could reshape how we build audio-visual eye. Instead of thinking of audio as a global label for a clip, we can think of it as a set of individual sources with persistent identities that evolve over time. That's a shift that could help robotics, hearing aids, and augmented reality.

Tom: That's a fitting place to stop. And let's not forget the authors: Zhi Zeng, Cheng Zhang, Zesheng Yang, Rendong Pi, and their collaborators from Xi'an Jiaotong University, Hong Kong Polytechnic, NUS, Central China Normal, and UC San Diego. Good paper, good discussion.

Jane: Agreed. And before we go, a quick note for our listeners: the field of omni-modal reasoning is moving fast, and this paper is one of the more concrete steps we've seen.

Tom: Thanks for being on the show, everyone. And thanks to our listeners. Next time, we'll have another paper to dig into. Until then, keep listening, and keep seeing.