Capability not equal to Interpretability: Human Interpretability of Vision Foundation Models

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Capability not equal to Interpretability".

Jane: The gist: Foundation models are consistently less interpretable than their supervised counterparts, and interpretability does not correlate with downstream task performance on any benchmark examined.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So we're talking about this paper today called "Capability not equal to Interpretability: Human Interpretability of Vision Foundation Models." It looks like it tackles a really big problem because these vision models are moving out of just research and into real, high-stakes applications. Jane Right. The core question they’re asking is, how interpretable are the features these leading vision models actually learned? Because right now, we don't have a reliable way to answer that reliably.

Lu: They set up a framework for measuring this interpretability using two psychophysics protocols: feature localizability and feature nameability. Tom Feature localizability is about whether an observer can predict where a specific feature fires on a new image, and nameability is if that observer can accurately describe what that specific feature actually represents.

Jane: That sounds like it could be useful for understanding how these models are actually seeing the world, not just how well they pass a test. Lu Exactly. And to make sure they compare different models fairly, they use sparse autoencoders to pull out these features and then a chance-anchored scoring function to put every model on the same scale.

Meng: I’m curious about the setup—how do you actually measure that nameability part? It sounds like something that requires human judgment. Tom They used this framework on six vision transformers, two supervised models and four foundation models: DINOv2, DINOv3, CLIP, and SigLIP.

Jane: That's a good spread. They collected more than fifteen thousand behavioral responses in total, and they analyzed thirteen thousand of those responses from the three hundred seventy-seven participants who passed their quality checks <ref:2605.20337#pg1,responses from the 377 participants>. Lu That’s a decent sample size to get some solid data on this topic.

Meng: So what did they find when they put all that data together? Did the foundation models turn out to be better or worse in terms of human understanding?

Tom: The headline finding is actually pretty sobering. They found that every foundation model tested produces features that are consistently less interpretable than those of the supervised baselines that came before. Jane Wait, so capability doesn't automatically equal better understanding then?

Lu: That’s right. The paper states clearly that downstream task performance does not predict this gap in interpretability. Tom So it sounds like the gap isn't just a trade-off for bigger models, which is something I always worry about when we ship things to production.

Jane: It really shifts the focus away from just checking if the model gets a better score on its job and towards understanding *how* it’s making those decisions. Lu They found that interpretability isn't one single thing, though. There are two properties that actually predict whether a model is more interpretable to humans: feature locality and coarse-grained semantic alignment with human similarity judgments.

Tom: So, the paper says that when you look at what’s driving this gap, it’s about where the features are located in the image and how well they line up with broad categories we already have in our heads. Jane That makes sense because those are things we intuitively look for when trying to understand a picture.

Lu: The paper points out that the locality of feature activations is the strongest predictor, correlating strongly with both localizability and nameability. Tom And coarse-grained semantic alignment also tracks interpretability, especially on tasks like the THINGS similarity dataset.

Jane: So what does this mean for someone who just listens to this show? It suggests that if you want a vision model to be understandable by a human, you need to train it so its features are both localized and organized around those large semantic categories.

Meng: From a deployment standpoint, if the gap between foundation models and supervised models is driven by something like this alignment, then we know what kind of training objective we should be pushing for. Tom It’s less about chasing bigger numbers and more about enforcing structure in the learned features.

Jane: And the paper gives an example with DINOv3, which has a training objective that explicitly promotes local features, suggesting that capability and interpretability don't have to trade off if you build locality into the training itself. Lu That’s a really encouraging counterexample in the work.

Tom: It confirms what we suspect: it’s not just about raw power; it’s about the internal structure of those learned representations that matters for human comprehension. Jane So, to wrap up this discussion on "Capability not equal to Interpretability: Human Interpretability of Vision Foundation Models," the paper argues that capability and interpretability are separate dimensions.

Lu: And they suggest a path forward by focusing training objectives that promote local feature activations and align those representations with the coarse semantic structure of human perception. Meng That gives us a concrete direction for model development, moving beyond just maximizing performance on benchmarks. Jane It moves the conversation from just "how smart is this model?" to "is this model organized in a way that humans can grasp?"

Conclusion: Tom: So we’re wrapping up on "Capability not equal to Interpretability: Human Interpretability of Vision Foundation Models." It boils down to this: just because a model can do a task really well doesn't mean we understand *why* it does what it does.

Jane: Right, so the authors are showing that these leading vision models, like DINOv3 and others, are actually less interpretable than the older supervised models that came before them.

Lu: They set up this framework with two tests for humans: localizability, which is about predicting where a feature fires on a new image, and nameability, which is about describing what that feature actually represents.

Meng: I'm still thinking about the numbers they used. They looked at six vision transformers—two supervised ones and four foundation models like DINOv2, DINOv3, CLIP, and SigLIP. That’s a pretty broad comparison set.

Tom: Yeah, and the key finding is that capability doesn't predict interpretability at all. The gap isn't about the model being less smart overall on benchmarks like ImageNet or COCO; it’s a structural problem with how the features are organized internally.

Jane: That means if we just keep making models bigger and better at classification, we aren't actually getting a more understandable representation of the world. It’s not just a performance issue for me; it’s an understanding issue.

Lu: Exactly. They found that two things actually predict interpretability: the locality of the feature activations, meaning they fire in specific spots on an image, and coarse-grained semantic alignment, where the features line up with broad categories we use to describe things.

Meng: So what does this mean for me when I’m trying to deploy a model? It tells me that I need to focus my training objectives on forcing those features to be localized and aligned with real-world categories, not just chasing higher accuracy scores.

Tom: That’s the actionable part. The paper suggests that training methods promoting local feature activations and aligning representations with how humans group things is what we need to prioritize. It points toward a recipe for building more graspable vision models, not just bigger ones.

Jane: So the main point is that we have two measurable ways to test if a model's internal representation makes sense to us, and right now, foundation models are failing on both counts compared to their supervised counterparts.

Lu: And they even suggest that some specific training objectives for certain foundation models, like DINOv3’s objective promoting locality, show that capability and interpretability don't have to trade off if you build locality directly into the training process.

Meng: That gives me something concrete to look into. If I can design a loss function that enforces those local feature activations, maybe I can get a model that’s both strong and explainable at scale.

Tom: Exactly. It shifts the focus from just optimizing for output accuracy to optimizing for internal structure, which is what actually matters when we move these things out of the lab and into real applications.

Jane: And this opens up a new way to evaluate AI models, moving beyond just testing them on standard tasks and looking at how their features behave in a way that mimics human perception.

Lu: It’s about getting those representations organized around the same high-level semantic structures that we already have in our heads. That connection is what matters for human understanding.