OmniSelect: Dynamic Modality-Aware Token Compression for Efficient Omni-modal Large Language Models

summary

The gist

The gist: OmniSelect, a training-free, modality-adaptive token pruning framework, dynamically selects appropriate compression strategies for multimodal inputs to maintain strong performance while

In short

OmniSelect is a training-free method for compressing long audio-video sequences in OmniLLMs. It dynamically selects compression strategies—Audio-Centric, Video-Centric, or Uniform—based on cross-modal relevance estimated by AudioCLIP. This adaptive pruning preserves model accuracy while significantly speeding up inference and reducing memory usage.

Key concepts

OmniSelect: A training-free framework that dynamically chooses the best way to compress multimodal inputs. It uses a two-stage approach: first, it decides whether to focus on audio or video based on relevance scores, and second, it prunes tokens within those groups adaptively.
AudioCLIP: A lightweight model used to estimate how relevant visual and audio information are to a given query. It calculates similarity scores between visual and audio embeddings, which helps OmniSelect categorize the input into Audio-Centric, Video-Centric, or Uniform pruning regimes.
Temporal Group Pruning Pipeline (TGP2): The second stage of the process that prunes redundant tokens within each temporal segment. For audio tokens, it uses attention scores to select salient ones; for video tokens, it uses cosine similarity to keep visually distinctive frames, ensuring important information is retained.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 18 episodes
Federated Learning — explained in 17 episodes
Retrieval-Augmented Generation (RAG) — explained in 16 episodes
Interpretability — explained in 15 episodes
Quantum Fisher Information (QFI) — explained in 15 episodes

This episode discusses

The paper

OmniSelect: Dynamic Modality-Aware Token Compression for Efficient Omni-modal Large Language Models · Read on arXiv

Morunliu Yang, Ruotao Xu, Le Li, Yue Wang, Jianxin Zhang, Juntao Li, Yihang Lou, Siwei Feng

Soochow University · Peking University

Omnimodal large language models (OmniLLMs) have recently gained increasing attention for unified audio-video understanding. However, processing long multimodal token sequences introduces substantial computational overhead, making efficient token compression crucial. Existing methods typically rely on fixed, modality-specific guidance, which fails to account for the varying importance of modalities across different queries. To address this limitation, we propose OmniSelect, a training-free, modality-adaptive token pruning framework that dynamically selects appropriate compression strategies for multimodal inputs. Specifically, we leverage a lightweight AudioCLIP model to estimate cross-modal relevance and categorize each input into three pruning regimes: Audio-Centric, Video-Centric, and Uniform pruning. Based on these relevance scores, OmniSelect further performs fine-grained token pruning within each temporal group, adaptively allocating pruning ratios to preserve informative tokens across modalities. By explicitly modeling modality preference and enabling dynamic strategy selection, OmniSelect effectively avoids the pitfalls of one-size-fits-all compression. Extensive experiments demonstrate that our method achieves efficient multimodal token reduction while maintaining strong performance, without requiring any additional training.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "OmniSelect: Dynamic Modality-Aware Token Compression for Efficient Omni-modal Large Language Models".

Jane: The gist: OmniSelect, a training-free, modality-adaptive token pruning framework, dynamically selects appropriate compression strategies for multimodal inputs to maintain strong performance while achieving inference speedup and memory reduction.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we’re talking about OmniSelect today. It’s titled "OmniSelect: Dynamic Modality-Aware Token Compression for Efficient Omni-modal Large Language Models." Jane, what’s the actual takeaway from that title for someone listening right now?

Jane: Well, it sounds like it’s about making those massive audio and video models smaller without losing too much quality. It’s not just cutting tokens by a fixed amount anymore; it sounds like it adapts based on what the model is actually looking at.

Tom: Exactly. It suggests a system that changes its compression strategy dynamically depending on the input, instead of using one fixed rule for everything. This is interesting because most existing methods use a uniform approach across modalities.

Lu: What I find compelling is the idea of explicitly modeling modality preference within the pruning process itself, which means it’s not just blindly cutting tokens based on size or position.

Meng: From a deployment standpoint, dynamic strategy selection sounds complicated to implement at scale. I need to know if this adaptation actually translates into a stable inference cost.

Lalam: Lalam here is confident that the core idea is about avoiding those pitfalls of fixed guidance because the importance of audio versus video changes depending on the query.

The paper's summary: Jane: So, what does OmniSelect actually do? Basically, it’s a training-free framework that uses AudioCLIP to figure out how important visual and audio information is for each temporal group in a video.

Tom: Right. It calculates similarity scores between the visual and audio representations using AudioCLIP to define pruning ratios for each group. That sets up three different types of pruning: uniform, video-centric, or audio-centric based on those scores.

Lu: The key mechanism is this two-stage strategy: first, using similarity to set the ratio, and then a temporal group pruning pipeline that actually prunes the redundant tokens within each frame.

Meng: So it’s not just one decision; it’s a calculation based on scores, followed by an actual token selection process in each time step. How does that actually translate to saving memory?

Lalam: It saves memory and speed by pruning intelligently, aiming to keep the tokens that are most informative across both modalities. The paper shows it achieves this without any new training required.

Jane: And they show impressive results on audio-visual understanding tasks, achieving competitive performance with existing training-free methods while cutting down the GPU memory usage.

The paper's improvements: Tom: Let’s talk about the specific improvements they propose. They move away from fixed, modality-specific guidance toward this dynamic selection mechanism based on those similarity scores.

Lu: They introduce a two-stage strategy where they first use AudioCLIP to define the ratio and then employ the Temporal Group Pruning Pipeline, TGP2, to perform fine-grained pruning within each temporal group.

Meng: The paper mentions using Top-K selection for audio tokens based on attention scores and Bottom-K selection for visual tokens based on cosine similarity matrices. Is that how they ensure they aren't just deleting random data?

Lalam: They use those specific methods—Top-K and Bottom-K—to select the most salient tokens, which is important because it targets visually distinctive tokens for vision and salient audio tokens for audio.

Jane: And they even introduce a mechanism where the threshold theta controls the transition between those different pruning strategies, suggesting you can tune how sensitive it is to modality gaps.

Conclusion: Tom: So to wrap up, OmniSelect is training-free, and it dynamically selects compression strategies based on calculated similarity scores to prune tokens across modalities without needing any additional training.

Jane: The big point here is that existing methods often rely on fixed guidance that doesn't account for how much audio or video matters in a specific context, which OmniSelect tries to fix by being adaptive.

Lu: It shows that by modeling modality preference explicitly, you can achieve efficient multimodal token reduction while maintaining strong performance on tasks like audio-visual understanding.

Meng: The efficiency gains they report are significant: a speedup of one point one nine times to one point three three times and memory reduction between two point six one gigabytes and two point eight one gigabytes when compressing about seventy percent of the audio and video tokens on the WorldSense benchmark <ref:2605.18041#pg3>.

Lalam: And they maintain high accuracy, retaining ninety-four percent to ninety-nine percent of the original full-token accuracy even with those aggressive compression settings.

Tom: So, OmniSelect presents a way to get substantial resource savings without sacrificing reasoning quality for audio and video tasks. That’s what we’ve covered on OmniSelect today.

More episodes

← Home