The Metagame of Interpretability and Meta-Attributions

arXiv:2605.06295 · cs.LG, cs.AI, stat.ML · Submitted 2026-05-07 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "The Metagame of Interpretability and Meta-Attributions".

Jane: We introduce METAGAME, a conceptual framework for quantifying second-order interaction effects of model explanations, which provides a principled method to decompose any first-order attribution into directional interaction terms.

Tom: First, who's behind it and why it matters.

Title and authors: Tom: So we've been looking at the paper "The Metagame of Interpretability and Meta-Attributions," which is really diving into how we can measure the second-order effects when explaining an AI model. It sounds like they're proposing a new way to look at those interactions that existing methods just don't capture well.

Jane: Exactly, Tom, and what struck me right away is their idea of treating the attribution method itself as a cooperative game and using the Shapley value to find these second-order interactions. It sounds like they are trying to give us a more principled way to break down any first-order explanation into these directional interaction terms.

Lu: That’s fascinating because it moves beyond just seeing which features are important individually and tries to quantify how one feature's importance affects the importance of another, which is a much richer way to see model behavior.

Meng: From an engineering standpoint, I’m wondering how this framework translates into actual system design; can we actually implement this kind of complex game-theoretic calculation without it taking forever?

Lalam: I think the most exciting part is that they show this works across different types of AI applications, which suggests it could be a really versatile tool for understanding how various models function.

Tom: Speaking of versatility, the paper lays out how these meta-attributions are not just some new numbers, but are directional extensions of existing interaction indices like the STII or SOP. It seems they've done some heavy lifting by showing these meta-attributions elegantly split classical set-based interactions into their precise components.

Jane: That directional decomposition is huge because it means we get to see exactly how the influence flows from one part of the input to another, rather than just getting a blended score for both features at once.

Lu: The paper shows specific mathematical proofs, like showing that for the Möbius transform, the directional scaling yields a precise split between pure effects and interactions involving sets larger than two elements. That level of detail in isolating those terms is what makes this framework compelling from a theoretical side.

Meng: I’m still focused on the practical side of things; if we use something like Meta-Shapley values, how much computational overhead are we looking at when running this on large language models?

Lalam: The paper actually addresses that by showing that for some specific applications, like ConceptAttention in diffusion transformers where there are twenty-four input tokens, a single sweep over coalition sizes can produce both diagonal Shapley values and every off-diagonal directional meta-attributions in seconds on a single H100 GPU.

Title and authors: Tom: That’s some solid efficiency information, Lu; the paper seems to have really thought about making this framework usable rather than just staying in the purely theoretical realm. It moves from concept to empirical demonstration pretty quickly.

Jane: I agree, and it's interesting how they connect these meta-attributions back to established indices like the Meta-IG acting as a directional variant of the SOP, which is helpful for anyone already familiar with those concepts.

Lu: The connection between their proposed meta-attributions and the existing interaction indices is where the real theoretical meat of "The Metagame of Interpretability and Meta-Attributions" lies, establishing them as these directional extensions.

Meng: So, if we look at the specific examples they use, like quantifying token interactions in instruction-tuned language models by using Meta-AttnLRP as Shapley values from text tokens into AttnLRP token attributions, that shows it’s not just abstract math.

Lalam: It really does show practical value because it lets us quantify those directional second-order effects in a way that was previously hard to see clearly in the results of methods like AttnLRP.

Tom: And that leads us into what they suggest for improvements, which is where this research gets really interesting for the future of interpretation tools. They point out that we could use this to quantify synergies and antisynergies between tokens, moving past just identifying important tokens individually.

Jane: Quantifying those synergies sounds like a huge step because it allows us to see how the presence of one token actively changes the importance of another, which is vital for understanding complex things like negation effects in language.

Lu: I think that ability to detect context-dependent shifts in focus based on these directional influences could lead to entirely new ways of debugging why an AI produces a certain output, moving from static explanations to dynamic interaction maps.

Meng: That would be powerful for debugging; if we can see exactly how a feature's importance is being amplified or suppressed by another, it helps isolate the source of error much better than just looking at the total attribution score.

Lalam: For cultural applications, I think this means we could build systems that understand complex social dynamics in text better because they would see how different concepts interact within a prompt structure.

Tom: Exactly; and then there's the point about rigorous interaction decomposition for model debugging, where they show how you can separate the pure effect of a feature from its directional interaction with others. This addresses that leakage problem that plagues many current methods.

Title and authors: Jane: That separation is key because it gives researchers a much clearer view into what’s due to genuine feature relevance versus what’s just an artifact of how features are grouped together during the explanation process.

Lu: The paper also touches on context-aware concept interpretation for multimodal models, suggesting we treat the entire set of concepts as a cooperative game to analyze interactions within complex scenes or generated images.

Meng: In vision-language tasks, if we can quantify the negative interaction between a text concept and a visual patch, that helps us build more robust systems that don't misinterpret how those two things relate to each other in the world.

Lalam: That directed interaction matrix for concepts and patches would give us a structural understanding of how multimodal inputs are synthesized, which is much better than just looking at them separately.

Tom: So, we’ve covered the core findings and now it’s time for Lu to weigh in on the broader implications of this work on AI interpretation methods. What's your take?

Lu: I see the implication as establishing a new hierarchy for interpretability, where meta-attributions become a fundamental tool for dissecting model explanations, allowing us to move from simple importance scores to understanding the underlying relational structure of how the model processes information.

Jane: That hierarchy is significant because it provides a principled way to decompose complexity without losing fidelity in the explanation itself.

Meng: I just want to make sure we hit reality; this framework needs to be computationally efficient enough for deployment, and while the paper shows good scaling for small games, we still need solid methods for larger, real-world model inputs.

Lalam: The ability to get these insights across instruction-tuned language models, vision-language encoders, and diffusion transformers means this framework has the potential to be a standard tool in many different AI research pipelines.

Tom: Absolutely; it’s not just one trick for one model type; it's a conceptual framework that applies across the board, which is exactly what makes "The Metagame of Interpretability and Meta-Attributions" so interesting.

Jane: So, to wrap up this segment, we see this paper provides a powerful mathematical structure for quantifying second-order effects through meta-attributions while demonstrating its applicability in various AI domains.

Lu: It really sets the stage for a deeper level of understanding regarding how model explanations function at a structural level.

Meng: We need to keep pushing on the implementation side to see if we can get these concepts into production systems efficiently.

Lalam: I’m excited to see how this framework evolves as more diverse AI models are developed and interpreted.

The paper's summary: Tom: So, to wrap up our first look at "The Metagame of Interpretability and Meta-Attributions," we've seen that this paper introduces a new conceptual framework for measuring how model explanations work, moving beyond just looking at which parts of an input matter most.

Jane: Exactly! The core idea is that instead of just getting a single score for an explanation, they propose treating the process of explaining a model as a cooperative game where you can find these second-order interactions.

Lu: That’s the big theoretical move, Jane; they are formalizing how to see those dependencies—how one feature's importance changes based on another feature's presence.

Meng: From my side, I’m thinking about the practical utility; if we can rigorously separate the direct effect of a token from its indirect influence on another token, that helps us debug model failures much more clearly.

Lalam: And for me, Lalam, this means we can finally start mapping out complex relationships within models in a way that’s genuinely useful for understanding how they process information.

Tom: Right! The paper shows these new meta-attributions are directional extensions of existing interaction indices like the ones we've seen before, which gives us a solid foundation to build on.

Jane: It’s about providing a principled way to decompose any first-order explanation into these specific directional terms, which is super helpful for understanding the mechanics underneath.

Lu: They prove that these meta-attributions hierarchically decompose into these smaller pieces, establishing them as a structured extension of what we already know about how features interact.

Tom: And the experimental results across language models and vision-language encoders show this framework isn't just theoretical fluff; it works in real AI systems.

Jane: That’s what I find really compelling; seeing these concepts applied to both text and vision models shows the versatility of this approach across different AI architectures.

Meng: So, the implication for us engineers is that we can start building tools that don't just tell us *what* is important, but *how* those important things influence each other.

Lalam: That directed understanding could really help in developing more nuanced systems that are better at reasoning and interpreting complex inputs in a cultural context.

Tom: It’s about moving from simple importance scores to a deeper structural understanding of how the AI actually reasons when it makes a decision.

Jane: And this structural view is exactly what makes these meta-attributions so powerful for analysis, because it captures the flow of influence between components.

Lu: This work really sets up a new hierarchy for interpretability, suggesting that second-order interactions are not just side effects but fundamental parts of the model's explanatory structure.

Tom: So, we’ve seen how this framework is mathematically rigorous and empirically tested across different AI domains, which opens up some seriously interesting avenues for future work.

Jane: And it’s exciting to think about what these new tools could mean for how we trust and understand the complex AI systems we use every day.

The paper's improvements: Tom: So, we’ve been digging into the core mechanics of "The Metagame of Interpretability and Meta-Attributions," and now we’re looking at what they suggest to make this framework even more useful for real-world AI applications.

Jane: They propose several improvements that focus on making these meta-attributions more actionable, rather than just theoretical concepts, which is really encouraging.

Lu: They suggest moving towards quantifying synergistic and antagonistic effects between tokens in language models, which means we can finally pinpoint exactly how one word's presence affects the importance of another.

Meng: That sounds powerful for debugging; if we can detect those specific interaction signals, it cuts through the noise and lets us see the actual flow of influence in a model’s decision-making process.

Lalam: For me, that ability to map out these complex relationships between concepts could significantly improve how we design and deploy multimodal AI systems in a way that's more aligned with human understanding.

Tom: They also suggest a more rigorous way to separate the pure effect of an individual feature from its directional interaction with others during debugging, which directly addresses a problem we see with current attribution methods.

Jane: That separation is crucial because it lets us distinguish between what a feature contributes on its own and what it contributes as part of a larger system.

Lu: They also propose context-aware concept interpretation for vision-language models, treating the entire set of concepts as a game to analyze how they interact within complex scenes or generated images.

Meng: If we can quantify the negative interaction between a text concept and a visual patch, that helps us build more robust systems that don't misinterpret how those two things relate to each other in the world.

Lalam: That directed interaction matrix for concepts and patches would give us a structural understanding of how multimodal inputs are synthesized, which is much better than just looking at them separately.

Tom: It’s about taking this from a mathematical tool to something we can actually use to build better, more transparent AI systems that don't just give us scores but tell us the story behind the score.

Jane: And these improvements suggest a path toward building systems that are not only accurate but also deeply interpretable, which is a big step forward for user trust.

Lu: The authors also point to future work involving higher-order interactions, suggesting we can build on this to analyze even more complex dependencies within the model structure itself.

Meng: I'm still focused on the computational feasibility of these improvements; we need to see if these richer interaction analyses can run efficiently enough for real-time deployment across large models.

Lalam: I’m really looking forward to seeing how this leads into better systems that can truly grasp and represent the world in a nuanced, interactive way.

Conclusion: Tom: So, to wrap up our discussion on "The Metagame of Interpretability and Meta-Attributions," we’ve seen that this paper provides a principled mathematical structure for quantifying second-order interaction effects through meta-attributions across various AI systems.

Jane: That's right! Essentially, they've given us a way to look past simple feature importance scores and see the directional influence between different parts of an explanation.

Lu: It really establishes a new hierarchy for interpretability, showing how second-order interactions are not just noise but fundamental components of how a model generates explanations.

Meng: From my standpoint as an engineer, the implication is that we can start building tools that provide much deeper insights into model behavior without having to rely on approximations or assumptions.

Lalam: This kind of directional understanding could really help us build more sophisticated AI that can reason about complex scenarios in a way that feels much more intuitive for people.

Tom: It’s clear this work is going to be very influential in how we approach model analysis moving forward, and it lays a solid foundation for future research.

Jane: It certainly does, and the applications they’ve shown across language models and vision encoders prove that this isn't just abstract math; it has real-world utility.

Lu: I think the creative possibilities here are vast; imagining how these meta-attributions could be used to design entirely new types of model architectures that inherently account for these directional dependencies is really exciting.

Meng: I’m still focused on the engineering hurdle: making sure that when we apply this, it remains computationally light enough to run on standard hardware without introducing significant latency into our production pipelines.

Lalam: And for me, seeing this framework applied to complex multimodal inputs suggests a future where AI systems can communicate and synthesize information in ways that are far richer than what we currently see.

Tom: So, as we conclude this segment on "The Metagame of Interpretability and Meta-Attributions," it’s been fantastic unpacking how these meta-attributions give us a clearer, more directional view of model explanations.

Jane: It really has been a deep dive into the mechanics behind understanding complex AI behavior.

Lu: This paper opens up so many avenues for theoretical exploration in how we model relational data within large neural networks.

Meng: I just hope we see this framework being implemented efficiently in production systems soon, because the potential for better debugging is huge.

Lalam: I’m really looking forward to seeing how this idea evolves into tools that help AI systems interact with the world in a much more meaningful way.

University of Warsaw · Centre for Credible AI, Warsaw University of Technology · Bielefeld University

cs.LG, cs.AI, stat.ML

Submitted: 2026-05-07

Updated: 2026-10-06

Comments: NeurIPS 2026. Code: https://github.com/credibleai/metagame

Code: https://github.com/credibleai/metagame

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 88/100

The gist: We introduce METAGAME, a conceptual framework for quantifying second-order interaction effects of model explanations, which provides a principled method to decompose any first-order attribution into

Key concepts

First-Order Attribution
This is the initial method used to explain a model's prediction by assigning importance scores to individual elements like words or image patches. It shows which parts of the input are most influential in the final output, but it doesn't show how these parts work together.
Meta-Attribution (e.g., $\phi_{j\to i}$)
These are directional extensions of standard interaction indices. They measure the specific influence one feature (j) has on the attribution score of another feature (i). This reveals the 'how' and 'in what direction' features cooperate to produce an explanation.
Shapley Value in METAGAME
The framework applies Shapley values, typically used in cooperative game theory, to the process of attribution itself. By treating the attribution method as a game, it extracts second-order interactions that show how features contribute synergistically or antagonistically to each other's importance scores.

Terminology

Summary

We introduce METAGAME, a conceptual framework for quantifying second-order interaction effects of model explanations, which provides a principled method to decompose any first-order attribution into directional interaction terms. This work matters because it establishes meta-attributions as a directional extension of existing interaction indices, offering rigorous hierarchical decomposition and practical insights across language models, vision-language encoders, and diffusion transformers.

How it works

The METAGAME conceptualizes the process by treating an arbitrary first-order attribution method, denoted as a function ϕi(f), as a cooperative game and applying the Shapley value to extract second-order interactions. For any first-order attribution ϕ(f) explaining a model f, the framework measures the directional influence of feature j on the attribution of feature i, denoted as meta-attribution φj→i(f), by treating the attribution method itself as a cooperative game and computing its Shapley value. Theoretically, this approach proves that attributions hierarchically decompose into meta-attributions, establishing these as directional extensions of existing interaction indices.

Key Theoretical Frameworks

The paper defines an interaction φj→i(f, x) as hierarchically efficient if the original first-order attribution is exactly preserved as a marginal sum of its pure individual effect φi→i(f, x) and its interactions with other features φj→i(f, x). The core mechanism involves fixing the value of feature xi and evaluating the marginal contribution of feature xj directly with respect to the first-order attribution ϕi(f, x), thereby isolating pure individual effects from interactions. This is formally defined as:

φj→i(f, x):= X S⊆[d] (1/(d − 1) · d−2 S) [ϕi(S ∪ i; f, x) − ϕi(S ∪ i; f, x)].

Directional Decomposition and Equivalence

The framework establishes several key equivalences between the proposed meta-attributions and existing interaction indices:

  1. The Meta-Shapley value is a directional variant of the STII, where ψSTII is decomposed into φMeta-SVj→i and φMeta-SV i→j.

  2. The Meta-IG acts as a directional variant of the SOP, where ψSOP is computed by summing the corresponding directional Meta-IG components: ψSOP i,j = φMeta-IG j→i + φMeta-IG i→j.

  3. The proof demonstrates that these meta-attributions elegantly split classical set-based interactions into their precise, directional components. For instance, for the Möbius transform mS(f, x), the directional scaling yields: φj→i(f, x) = 1/2 mS(f, x) + X T ⊃ S:T>2 3T(T+1)mT (f, x).

Empirical Applications

The METAGAME framework is demonstrated across three diverse applications:

  1. Quantifying token interactions in instruction-tuned language models by computing Meta-AttnLRP as Shapley values from text tokens into AttnLRP token attributions. This highlights directional second-order effects.

  2. Explaining cross-modal similarity in vision–language encoders by using MetaCLIP-2 to compute meta-attributions, showing that the proposed method is the most faithful to interpret SigLIP-2.

  3. Interpreting text-to-image concepts in multimodal diffusion transformers via Meta-ConceptAttention, which computes a Shapley value of ConceptAttention on the METAGAME defined by input tokens.

Computational Efficiency and Scaling

The framework is designed to be computationally feasible. For small METAGAMES (d < 20), the Shapley value can be computed exactly in one vectorized pass. For larger games, the authors recommend using SOTA approximators from the shapiq library, such as Monte Carlo sampling or regression-based approximators, which better focus computational budget when d is large. Furthermore, for ConceptAttention in diffusion transformers (d ≤ 24), a single sweep over coalition sizes produces both diagonal Shapley values and every off-diagonal directional meta-attributions in seconds on a single H100 GPU.

Limitations and Future Extensions

The theory is limited by assuming baseline masking/imputation—the most principled approach to removing tokens/features from machine learning model inputs. Empirically, while exact Shapley values are feasible for small games, approximations are necessary for larger ones. The framework can be extended to other masking strategies (e.g., feature marginalization) and higher-order interactions by computing higher-order Shapley interaction indices directly on the METAGAME.

Improvements for AI systems

Based on the METAGAME framework presented in the paper, here are specific improvements that can be made to AI systems across different domains:

) 1. Quantifiable Synergies and Antisynergies in Token Interactions (Language Models)

The improved system can move beyond simply identifying which tokens are important by quantifying their meta-attributions (directional influence on the attribution of another token).

  • Specific Capability: For an instruction-tuned LLM, the system can detect complex synergistic or antagonistic relationships between token pairs that a first-order method misses. For example, it can explicitly quantify how the presence of 'honey' positively influences the model's attention to 'butter' in a prompt, even if both tokens have low individual importance on their own.

  • Specific Output: It will output a ranked list of token interactions (e.g., token A has a strong positive influence on token B’s importance, or removing token C nullifies the negative signal between A and B). This allows for more nuanced understanding of model comprehension, such as detecting negation effects or context-dependent shifts in focus (as demonstrated in Figure 10).

) 2. Rigorous Interaction Decomposition for Model Debugging (General Attribution Methods)

The system can provide a mathematically rigorous way to separate individual feature effects from interaction effects, which is currently a major limitation of serial methods like Integrated Gradients or standard Shapley values.

  • Specific Capability: When debugging why a model made an incorrect prediction, the system can decompose the total attribution score into three distinct components: the pure effect of a feature (e.g., Feature X contributes Y), and its directional interaction with other features (e.g., Feature X's presence increases the importance of Feature Z by factor W).

  • Specific Output: It will generate a hierarchical breakdown showing exactly how much of the overall prediction score is due to individual feature relevance versus higher-order joint effects, preventing interaction leakage observed in current methods.

) 3. Context-Aware Concept Interpretation for Multimodal Models (Vision-Language Encoders and Diffusion Transformers)

The system can overcome the context-dependence issue inherent in methods like ConceptAttention by treating the entire set of concepts as a cooperative game, allowing for robust analysis of interactions within complex scenes or generated images.

  • Specific Capability: In vision-language tasks, it can quantify how text concepts interact with visual patches (e.g., the interaction between the concept 'black dog' and the visual patch of the 'hydrant' is strongly negative). For diffusion models, it can analyze cross-modal concept interactions in generated images (e.g., the interaction between concept 'car' and concept 'street' is highly positive in this specific image generation).

  • Specific Output: It will produce a directed interaction matrix for concepts and patches, revealing the underlying structural relationships that govern how multimodal inputs are synthesized, leading to more faithful explanations of complex scene understanding.

) 4. Efficient Single-Pass Second-Order Analysis (Computational Efficiency)

The framework enables the computation of second-order interactions using only a single forward pass by exploiting cached activations, drastically reducing computational overhead compared to methods requiring repeated sampling or complex architectural modifications.

  • Specific Capability: It can perform full second-order interaction analysis (Meta-Shapley values and Meta-IGs) on large models or high-dimensional feature spaces (like vision patches) with computational complexity that scales better than combinatorial explosion, making it feasible for real-time applications.

  • Specific Output: Real-time, comprehensive interaction maps for large vision models or language prompts, ensuring that the interpretability process does not introduce prohibitive latency.

Abstract

How can an arbitrary attribution method be generalized from first principles to capture interactions? We answer this with the metagame, a conceptual framework for quantifying second-order interaction effects of model explanations. We cast the attribution value ϕ i of feature i as a cooperative game among the other features and compute its Shapley value, which measures how much feature j influences the attribution of i, yielding the directional meta-attribution φ j to i. By decomposing attribution itself rather than the model directly, meta-attributions extend any gradient- or attention-based method to interactions, uniting removal-based perturbations with model internals. Theoretically, we prove that meta-attributions sum to the first-order attribution they explain, a hierarchical decomposition that Shapley interactions and integrated Hessians turn out to perform implicitly. Empirically, we demonstrate that meta-attributions deliver insights across diverse interpretability applications: (i) quantifying token interactions in instruction-tuned language models, (ii) explaining cross-modal similarity in vision-language encoders, and (iii) interpreting text-to-image concepts in multimodal diffusion transformers.

Sources

Related papers