Open-Vocabulary Audio-Visual Event Localization via Complex-Valued Fusion

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Open-Vocabulary Audio-Visual Event Localization via Complex-Valued Fusion".

Jane: The gist The dominant pipeline uses a frozen multimodal foundation model (e.g.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're looking at this paper today titled Open-Vocabulary Audio-Visual Event Localization via Complex-Valued Fusion by Praveen and his team. Basically, they're tackling how to label video segments with events, even classes they never saw during training.

Jane: That’s right, and the main idea is that instead of using a standard way to combine visual information and audio information to pick an event class, they use complex numbers for that fusion.

Lu: It's interesting because they are replacing the existing scalar fusion rule—the one that just takes a geometric mean or a weighted average—with something much richer called a complex-valued neural network, or CVNN.

Meng: So instead of getting one number from the visual similarity and another from the audio similarity, they are creating two complex similarities for every segment against every class: visual–text and audio–text.

Tom: Exactly, and then they compute these two cosine similarities for each segment against each candidate class name. That feeds into their core idea of using this complex-valued neural network to learn how to fuse them.

Jane: What makes this approach different is that they treat the standard representation of each modality as the real part of a complex value, and they use a paired companion stream to supply the imaginary part.

Lu: For the visual side, the real part comes from the RGB content itself, but they add an imaginary part using an iHSV transform of that RGB frame before feeding it into a frozen vision encoder.

Meng: And for audio, it's symmetrical; the real part is taken from the log-mel magnitude content from a frozen audio encoder, and they use a CycleGAN-translated phase spectrogram to supply the imaginary part, which also feeds into that same frozen audio encoder.

Tom: They actually have two main ways to set this up: either a two-stream fusion where they package both modalities into one complex channel—real for visual similarity and imaginary for audio similarity—or a four-stream extension.

Jane: That four-stream extension is where they enrich each modality with those paired companion streams, and they stack those as two complex channels into the same ComplexCNN structure.

Paper summary: Lu: The ComplexCNN itself uses five layers, where the input to any layer is treated as a complex feature map x equals xR plus iX I, and the learnable kernel W equals WR plus iWI >

Meng: And they define a complex convolution operation where Re(W star x) equals WR star xR minus WI star xI, and Im(W star x) equals WR * xI plus WI * xr, which is a real two-D convolution over the time and class grid <ref:2610.11846#pg1>.

Tom: After those five layers of ComplexCNN, they get this final output magnitude, which they use to get per-segment class scores P hat t in R T times C plus one.

Jane: These scores are then min–max rescaled along the class axis to give you the final per-segment class scores. That's how the model spits out its decision for each segment and each possible event category.

Lu: Now, looking at what they trained, they kept things frozen, which means modules like the ImageBind encoders and the CycleGAN generator are trained offline.

Meng: The training only focuses on the trainable components, which include four temporal-attention blocks—Φv, ΦI v, Φa, and Φφ—and that ComplexCNN itself.

Tom: The objective for training is a per-segment cross-entropy loss L equals minus sigma log P hat t y t >

Jane: So the paper shows that the four-stream complex model achieves state-of-the-art results on both open set audio–visual event localization benchmarks, reaching sixty-six point five/fifty-nine point one/fifty-four point one percent accuracy for segments, F1 score for segments, and Event F1 score on the open split of OV-AVEBench.

Lu: And when they look at the isolated contributions by doing separate ablations, they show that complex parameterization is the main driver of improvement on both datasets.

Meng: Specifically, real to complex at two streams yields a gain of seven point nine Event F1 on OVAVEBench > <ref:2610.11846#pg1>

Tom: Adding those auxiliary streams—the iHSV for vision and the CycleGAN-phase for audio—adds a smaller, complementary gain, lifting unseen Event F1 by zero point eight on OV-AVEBench >

Paper summary: Jane: The paper also notes that the CycleGAN translation step for audio phase actually outperforms using raw IF spectrograms, improving every metric on OV-AVEBench by one point eight/two point four/two point two for accuracy, segment F1, and event F1 >

Lu: They also compared this against AVEOV and found the CycleGAN translation for audio phase improved every metric by one point nine on OV-AVEBench and tied on Segment F1 on AVEOV > <ref:2610.11846#pg1>

Meng: The authors conclude that the four-stream approach outperformed the two-stream approach on both datasets, although they admit its complexity might lead to lesser cross-dataset transfer capabilities > <ref:2610.11846#pg2>

Tom: What this means for us is that complex fusion is clearly the dominant element here, but those auxiliary streams add a smaller boost.

Jane: So if you think about what this does for someone who only listens to the show, it means we’re moving towards systems that can identify events in video and audio they weren't specifically trained on, which is key for open-vocabulary applications.

Lu: The paper proposes two layered architectural contributions: complex-valued fusion and a four-stream extension >

Meng: It also points out that this method is competitive with the dedicated closed set method AVSGN even when restricted to the closed set regime >

Tom: This work establishes a new state of the art on both OV-AVEL benchmarks, showing that complex parameterization is what gives them the lead in these open-vocabulary settings.

Jane: It confirms that complex fusion and auxiliary streams are complementary ingredients rather than substitutes when you compare them against real fusion at matched parameters >

Lu: The paper provides code available at https://github.com/AnirudhPraveen/Complex-Valued-OV-AVEL >

Meng: So, if we look forward, the authors suggest that the four-stream approach might be too complex for some scenarios regarding cross-dataset transfer > <ref:2610.11846#pg2>

Tom: It’s a trade off, really; you get these high accuracy numbers, but you might sacrifice some versatility when moving the system to a totally different set of data.

Jane: Overall, Open-Vocabulary Audio-Visual Event Localization via Complex-Valued Fusion shows that complex modeling provides richer representations than traditional real-valued networks in this specific multimodal fusion task >

Conclusion: Tom: So we're wrapping up this look at "Open-Vocabulary Audio-Visual Event Localization via Complex-Valued Fusion." Basically, they took standard video and audio data and fused them using complex numbers to identify events that weren't in their original training set.

Jane: That's right, the core idea is using a complex network—a CVNN—to combine visual and audio information instead of just using simple addition or averaging of real numbers.

Lu: They use this approach to create two different types of similarity scores: one for the visual part and one for the audio part, both operating in this complex space.

Meng: And they enrich each input with extra streams—like an iHSV transform for the picture and a CycleGAN phase spectrogram for the sound—to give those complex numbers more detail to work with.

Lalam: From a model perspective, it shows that when you feed richer, paired information into a complex structure, the resulting representations become much more nuanced for open-vocabulary tasks.

Tom: The authors found this four-stream setup gets them state-of-the-art results on these audio visual event localization benchmarks.

Jane: They showed that the four stream approach outperformed the two stream version across both datasets, meaning adding those extra streams actually helps the system perform better overall.

Lu: But they also noted that this complex model, while powerful for finding new events, might be a bit too heavy or complex for some kinds of cross-dataset transfers.

Meng: So the practical reality is that you get high accuracy on unseen events, but you have to consider the complexity when you try to move that system to a completely different type of data.

Tom: It really highlights how much more information we can pack into these fusion models by using complex math instead of just standard real-valued calculations.

Jane: It suggests that for things like event localization, moving from simple real-number fusion to complex representations opens up a whole new way to interpret what we're seeing and hearing in the world.

Lu: And this opens the door for thinking about how other multimodal tasks could benefit from this type of complex feature learning.