From What to Which: Decoding Modifier Grounding in Frozen MLLMs

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "From What to Which".

Jane: The gist: instance-level grounding is distributed across contextualized token representations rather than being confined to a single modifier token, and this phenomenon can be decoded using OTTER,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: Now we're moving into the conclusion of "From What to Which: Decoding Modifier Grounding in Frozen MLLMs." We've talked about how this paper investigates whether frozen MLLM representations contain decodable grounding information for referred instances across generated tokens, extending that concept to modifiers like attributes and spatial expressions.

Jane: The core claim here is that the what-to-which problem, shifting from object-level to instance-level grounding, can be solved by looking at these modifier tokens within frozen MLLM representations.

Lu: They address the gap left by previous work where contextual embedding methods showed grounding information exists at the answer level but didn't look at individual generated tokens.

Meng: So what they are doing is moving the analysis from just looking at a final answer to decoding signals directly from every token that makes up the generated phrase.

Tom: They introduce OTTER, a lightweight supervised probe using Optimal Transport to align these generated tokens with visual regions and produce compact grounding maps, without changing the frozen MLLM.

Jane: It’s important because it shows that instance relevance is distributed across contextualized token states rather than being stuck in just one modifier token.

Lu: Specifically, they show that instance-discriminative visual info can be decoded from modifiers with the strongest evidence for spatial terms, but also from contextual object nouns themselves.

Tom: And they confirm that this recovered grounding remains informative even when you introduce context perturbations, which means it's robust.

Jane: It extends to free generation and cross-dataset transfer, which is a significant step because it shows the learned geometry transfers to unconstrained settings.

Meng: From an engineering standpoint, this means we can build systems that rely on these internal representations without needing expensive fine-tuning for every new referring expression task.

Lu: They conclude that the transition from what to which is better understood as a process of progressively refined, distributed instance grounding across an expression.

Tom: It's about understanding how the context flows through the entire generation process to resolve referential ambiguity in a more detailed way.

Conclusion: Jane: So wrapping up with the conclusion for "From What to Which: Decoding Modifier Grounding in Frozen MLLMs," we're looking at the title and authors. The researchers are Corradini, Gallegati, Genovese, and Murino from IIT.

Tom: It really boils down to this idea that instance-level grounding isn't something you isolate in one token; it’s woven throughout the whole contextual representation.

Lu: Think about it like this: instead of just looking at the noun "banana," you look at how the word "yellow" and "left" guide your attention across all those tokens to pinpoint exactly which banana in a scene is meant.

Meng: It means that for future AI applications, we don't need perfect object localization for every single generated phrase; we can rely on the internal structure of a frozen model to decode the instance reference.

Tom: It suggests that understanding this transition from what to which happens through a process of progressively refined, distributed instance grounding across an expression.

Jane: So when we think about how these large language models will interact with visual scenes, we should focus on decoding this distributed signal rather than just focusing on the object itself.