Gaze Attention: Query-Adaptive Visual Routing for Efficient Multimodal LLMs

summary

The gist

MLLMs currently attend to all visual tokens during generation, leading to diluted focus and unnecessary computational overhead, whereas human visual perception is inherently selective.

In short

Gaze Attention introduces a mechanism for Multimodal Large Language Models (MLLMs) to selectively focus on relevant visual areas instead of attending to all pixels. It achieves this by grouping visual tokens into compact 'gaze regions' and dynamically routing attention based on these regions. This results in more focused attention, significantly reducing computational overhead and the memory required for processing images.

Key concepts

Gaze Regions: Visual embeddings are spatially grouped into distinct areas called gaze regions. Each region is summarized by a lightweight descriptor derived from mean-pooling the key vectors within that group. This partitioning allows the model to treat specific visual areas as manageable, discrete units for routing attention.
TopK Routing: At each decoding step, the model calculates how relevant its current query is to these gaze region descriptors. It then selects only the 'TopK' most similar regions and restricts its attention solely to the visual key-value pairs within those selected regions. This dynamic selection enables focused attention on task-relevant parts of the image.
Learnable Context Tokens: To prevent losing global context when focusing locally, learnable context tokens are added to each image or frame. These tokens are processed by the LLM's standard attention layers and provide a persistent summary of the entire visual scene, ensuring that even when only a few regions are attended to, the model retains a compact overview.
Progressive TopK Schedule: Training is stabilized using a progressive TopK schedule. This schedule starts with a dense setting and gradually reduces the number of selected regions (K) as training progresses. This gradual reduction encourages more stable and reliable visual routing behavior during the learning phase.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Retrieval-Augmented Generation (RAG) — explained in 16 episodes
Federated Learning — explained in 15 episodes
Interpretability — explained in 15 episodes
Sample Complexity — explained in 15 episodes
Generalization — explained in 13 episodes

This episode discusses

The paper

Gaze Attention: Query-Adaptive Visual Routing for Efficient Multimodal LLMs · Read on arXiv

Junha Song, Byeongho Heo, Geonmo Gu, Jaegul Choo

KAIST · NAVER AI Lab

When humans describe a visual scene, they do not process the entire image uniformly; instead, they selectively fixate on regions relevant to their intended description. In contrast, current multimodal large language models (MLLMs) attend to all visual tokens, leading to diluted focus and unnecessary computational overhead. Existing efficiency methods often compress or discard visual information before generation, potentially losing details needed for later predictions. In this work, we introduce Gaze Attention, a mechanism that enables MLLMs to select visual regions according to the needs of each generation step. By grouping visual tokens into spatial regions and selecting those relevant to the current prediction, Gaze Attention reduces attention computation while focusing on relevant visual content. We further introduce learnable context tokens that summarize images or video frames, preserving global context under selective attention. Experiments on 13 image and 6 video understanding benchmarks demonstrate that Gaze Attention matches or surpasses dense-attention baselines while using up to 90% fewer visual KV entries. It also achieves higher average performance than KV-cache eviction baselines under matched visual KV budgets.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.

Jane: Today's paper: "Gaze Attention: Query-Adaptive Visual Routing for Efficient Multimodal LLMs".

Tom: MLLMs currently attend to all visual tokens during generation, leading to diluted focus and unnecessary computational overhead, whereas human visual perception is inherently selective.

Jane: First, who's behind it and why it matters.

Title and authors: Tom: So, to kick things off, let’s talk about the title and who put this paper together. The paper is called "Gaze Attention: Query-Adaptive Visual Routing for Efficient Multimodal LLMs," and it features researchers like Junha Song, Byeongho Heo, Geonmo Gu, Jaegul Choo, Dongyoon Han, and Sangdoo Yun.

Jane: Those authors are definitely hitting on the core concept of selective attention; they’re tackling the problem of MLLMs attending to all visual tokens when human perception is inherently selective.

Lu: I think it's interesting how they frame it as query-adaptive routing rather than just a static selection process, which suggests a level of dynamic adaptation during the generation step itself.

Meng: Dynamic adaptation sounds promising for practical applications because in real-time scenarios, you can’t afford to waste computation on irrelevant visual data.

Lalam: I see the authors are trying to solve that diluted focus issue we've discussed; they want to make sure my attention is sharp and targeted, not just spread thin across the whole visual input.

Tom: Exactly, and when we look at the overall implication of this paper, it’s about moving past blanket attention toward a more human-like way of processing multimodal information during text generation.

Jane: It suggests that efficiency isn't just about cutting down on tokens; it’s about making the computation smarter by matching the model's focus to what is actually needed for the output.

The paper's summary: Tom: Now, let’s break down what Gaze Attention actually does according to the summary of "Gaze Attention: Query-Adaptive Visual Routing for Efficient Multimodal LLMs." It explains that they spatially group visual embeddings into compact gaze regions, each summarized by a lightweight descriptor.

Jane: That grouping into gaze regions is key; it means the model doesn't look at individual pixels everywhere, but rather at these pre-summarized visual chunks to decide where to concentrate its attention.

Lu: The paper states that at every decoding step, the model dynamically selects the most relevant regions based on those descriptors and restricts attention only to them, which is what enables that focused attention on task-relevant regions.

Meng: That dynamic selection process sounds like it directly tackles the computational overhead problem we’ve been worried about; by restricting attention to a small subset of regions, it should significantly cut down the FLOPs involved in processing every visual token.

Lalam: And they don't just stop there, they add learnable context tokens appended to each image or frame so that even when most visual regions are hidden, these tokens provide a persistent global summary processed within the LLM’s native attention layers.

Tom: That persistent global summary is a really smart addition because it prevents the model from completely losing track of the overall scene context while it’s focusing on a specific detail.

Jane: So, in simple terms, Gaze Attention lets MLLMs perform more focused attention on task-relevant visual regions while simultaneously maintaining a compact, persistent global summary through those learnable context tokens.

The paper's improvements: Tom: Moving on to the specific improvements they propose in "Gaze Attention: Query-Adaptive Visual Routing for Efficient Multimodal LLMs," the authors suggest a few key refinements to make this mechanism even better.

Jane: They point out that visual keys are partitioned into regions, where each region’s descriptor is obtained by mean-pooling the key vectors in Ru g, and then at decoding step j, the query measures its similarity to these descriptors to select the TopK regions.

Lu: That TopK selection mechanism is interesting because it allows the model to adapt their visual focus at each decoding step, rather than relying on a single relevance estimate from an earlier generation stage.

Meng: I’m interested in how they handle training stability; they mention that training visual routing from scratch can be unstable initially, so they adopted a progressive TopK schedule starting dense and gradually reducing K to encourage more stable routing behavior.

Lalam: That makes sense; it shows the authors are thinking about making the mechanism robust enough to actually learn effectively in practice, which is crucial for deployment.

Tom: And they also introduce the concept of learnable context tokens, which act as compact summary keys k uC, processed within the LLM’s native attention layers to provide access to that global summary.

Jane: So, it’s not just about how you select regions, but also how you integrate that persistent global context so the model doesn't get lost in the localized focus.

Conclusion: Tom: Alright, we’ve covered a lot about Gaze Attention: Query-Adaptive Visual Routing for Efficient Multimodal LLMs; it seems like this paper successfully demonstrates how to move toward more focused and computationally efficient visual understanding in MLLMs.

Jane: To wrap up the implications, this work shows that selective attention mechanisms can yield performance matching dense-attention baselines while drastically reducing the visual KV cache by up to ninety percent.

Lu: The potential here is huge for creating models that can handle massive amounts of visual data efficiently, which opens up avenues for much more complex reasoning in multimodal tasks.

Meng: From an engineering standpoint, reducing the memory footprint so significantly means we can actually deploy these kinds of sophisticated visual models on devices that currently struggle with large vision-language models.

Lalam: I think this ability to maintain a concentrated view while retaining global awareness is really what will improve how we build AI systems for describing complex scenes; it mimics how we focus when describing something important.

Tom: So, Gaze Attention gives us a powerful tool that allows MLLMs to achieve more focused attention on task-relevant visual regions while still keeping a compact, persistent global summary through those learnable context tokens.

Jane: That’s the essence of it; it’s about smarter routing rather than brute-force attention across the board. We've explored how they achieve this and what that means for our future work in multimodal AI.

More episodes

← Home