Decoding Alignment without Encoding Alignment: A critique of similarity analysis in neuroscience

summary

The gist

The gist Decoding approaches are widely used in neuroscience and machine learning to compare stimulus representations across neural systems, such as different brain regions, organisms, and deep

In short

Decoding metrics like RSA and CKA compare neural representations but fail to capture how neurons are organized internally (encoding topology). The study shows these metrics remain unchanged even when the internal functional organization is manipulated. This means similarity in decoding output does not guarantee similarity in underlying computation or structure.

Key concepts

Decoding Metrics: These are standard tools used to compare neural representations across different systems, such as brain regions or AI models. They look at how well two networks produce the same output for a given input stimulus, but they ignore the internal functional organization of the neurons themselves.
Encoding Manifold: This concept treats neurons as points in a space defined by their responses to various stimuli. The global shape or topology of this 'point cloud' reflects how functionally related or organized the neurons are within a population, such as forming distinct clusters or a smooth continuum.
Gromov-Wasserstein Distance (GW): This is a proposed complementary metric that measures the distance between two encoding manifolds. Unlike decoding metrics, GW looks at how different the internal functional organization (the topology) is between two populations, providing insight into structural differences that decoding metrics miss.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Federated Learning — explained in 17 episodes
Retrieval-Augmented Generation (RAG) — explained in 16 episodes
Interpretability — explained in 15 episodes
Quantum Fisher Information (QFI) — explained in 15 episodes
Sample Complexity — explained in 15 episodes

This episode discusses

The paper

Decoding Alignment without Encoding Alignment: A critique of similarity analysis in neuroscience · Read on arXiv

Johannes Bertram, Luciano Dyballa, T. Anderson Keller, Savik Kinger, Steven W. Zucker

School of Science & Technology IE University · The Kempner Institute for Natural and Artificial Intelligence Harvard University

RSA and CKA are standard metrics for comparing neural representations across brain regions, organisms, and deep learning models. We demonstrate a fundamental weakness: these decoding-based metrics are insensitive to encoding manifold topology---the internal functional organization of a neural population. In a controlled MNIST experiment, RSA, CKA, and Procrustes R squared remain statistically unchanged when encoding topology is causally manipulated via an auxiliary clustering loss, while the two model populations differ significantly in attribution patterns, weight-graph assortativity, and out-of-distribution robustness. Across biological systems and machine learning models, similar decoding behavior can arise from small, non-representative subpopulations, and alignment metrics are insensitive to encoding manifold topology even when it is fundamentally altered. These findings bear directly on mechanistic interpretability: standard alignment metrics cannot distinguish whether two networks share the same computational circuits or merely produce indistinguishable aggregate outputs. We propose encoding manifolds and Gromov-Wasserstein distance as complementary diagnostics for any decoding-based similarity claim, and provide a Neural Manifold Explorer tool at https://johannesbertram.github.io/FNN Manifolds/index.html for further experimentation.

Transcript

Introduction to the show: ident: Genomics Radio. Generated commentary on the latest computational biology and genomics papers.

Ines: Today's paper: "Decoding Alignment without Encoding Alignment".

Marcus: The gist Decoding approaches are widely used in neuroscience and machine learning to compare stimulus representations across neural systems, such as different brain regions, organisms, and deep learning models.

Ines: First, who's behind it and why it matters.

Paper summary: Ines: So, let's go over what the paper claims about this critique of similarity analysis. They start by saying that decoding approaches are used widely in neuroscience and machine learning to compare stimulus representations across different systems like brain regions or deep learning models, using metrics such as Representational Similarity Analysis RSA and Dynamic Similarity Analysis DSA.

Marcus: The main thesis they put forward is that assuming representational geometry is representative of the whole neural population is misleading because the representation might actually be shaped by a very small subset of neurons.

Ines: They show this by setting up a controlled experiment in MNIST, where they manipulate encoding topology causally using an auxiliary clustering loss, and find that RSA, CKA, and Procrustes R2 remain statistically unchanged when encoding topology is manipulated.

Yuki: The paper demonstrates that the two model populations differ significantly in attribution patterns and weight-graph assortativity even when the decoding behavior looks the same to these metrics.

Marcus: They also show that across biological systems and machine learning models, this similar decoding behavior can arise from small, non-representative subpopulations of neurons.

Ines: The paper highlights a critical limitation: alignment metrics operate on the same population-level distance matrices, so they inherit this blind spot where they cannot distinguish between two networks that have different internal functional organization.

Yuki: To address this, the authors introduce the complementary encoding paradigm, which characterizes how neurons are organized globally in terms of their responses to a set of data.

Marcus: This encoding manifold represents neurons as points in stimulus-response space, and each point on that manifold is a neuron; nearby neurons are similarly tuned.

Ines: The global topology of this neural point cloud—whether it forms discrete clusters or a smooth continuum—directly reflects the functional architecture of the population, independently of any downstream readout.

Conclusion: Yuki: So to wrap up, the authors argue that decoding metrics alone are insufficient because they treat the neural population as a monolithic unit when function might be distributed heterogeneously.

Marcus: They use their controlled experiment in MNIST to show that you can replicate full population decoding performance with only a small fraction of neurons under specific selection strategies.

Ines: And then they show that alignment metrics inherit this blind spot, meaning they cannot tell if two networks share the same computational circuits or just produce indistinguishable aggregate outputs.

Yuki: They suggest that encoding and decoding manifold topology can be independently manipulated to provide causal evidence that these metrics are insensitive to internal functional organization.

Marcus: So the practical takeaway is to use encoding manifolds and Gromov-Wasserstein distance as complementary diagnostics when you're comparing neural systems or deep learning models.

Ines: They recommend this approach because it helps us move beyond just looking at decoding scores and gives us a way to measure the actual internal functional organization of the population.

More episodes

← Home