Stimulus symmetries can confound representational similarity analyses

arXiv:2605.21324 · q-bio.NC, cs.LG · Submitted 2026-05-20 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Genomics Radio. Generated commentary on the latest computational biology and genomics papers.

Ines: I'm Ines, and with me are Marcus and Yuki, guest researcher.

Marcus: Today's paper: "Stimulus symmetries can confound representational similarity analyses".

Ines: Stimulus symmetries can confound representational similarity analyses because functionally equivalent representations related by stimulus symmetries can possess qualitatively different representational geometries, leading to distinct Representational Similarity Matrices (RSMs).

Marcus: First, who's behind it and why it matters.

Paper summary: Ines: So this paper, "Stimulus symmetries can confound representational similarity analyses," really tackles how we interpret similarity scores when the input data has inherent symmetries. The main idea seems to be that functionally equivalent representations related by these stimulus symmetries can end up having different geometric properties in representation space, which means their Representational Similarity Matrices or RSMs won't match up as expected <ref:2605.21324#pg0>.

Marcus: That sounds like a real problem for us when we try to use these summary statistics to compare different neural codes across different datasets; if the underlying data structure dictates the symmetry, then the similarity metric itself gets biased <ref:2605.21324#pg0>.

Yuki: From a population genetic perspective, this hints at how subtle variations in stimulus presentation or environmental factors might lead to functionally identical neural states that are structurally distinct when viewed through a rigid mathematical lens, which mirrors complexities we see in evolutionary history <ref:2605.21324#pg0>.

Ines: Exactly, and the paper claims that this means we can't just assume that if two representations are functionally equivalent because of a stimulus symmetry, their RSMs will be identical; instead, they can reflect qualitatively different geometries <ref:2605.21324#pg0>.

Marcus: So it’s not just about the inputs being symmetrical; it's about how those symmetries manifest in the resulting neural encodings and whether our comparison tool is sensitive to that manifestation <ref:2605.21324#pg1>.

Yuki: It suggests that what we perceive as functional equivalence at a neural level might be masked or distorted if the representation space itself isn't properly anchored relative to the symmetry group acting on the input manifold <ref:2605.21324#pg1>.

Ines: And they formalize this by looking at gauge invariance; they introduce a setting where data lives on a latent space Z acted upon by a compact group G, and they show that for an RSM to be gauge-invariant, the encoding has to be an orthogonal linear representation of G <ref:2605.21324#pg1>.

Marcus: That brings up the issue of how we define that orthogonality when dealing with non-linear systems; if the encodings aren't naturally related by a simple rotation in representation space, then that gauge invariance condition becomes very strict <ref:2605.21324#pg1>.

Yuki: It makes me think about how we model population dynamics where symmetries exist; if the underlying genetic structure imposes symmetries on phenotype expression, we need to account for those transformations when analyzing similarity <ref:2605.21324#pg0>.

Paper summary: Ines: They then move into showing that while the encodings themselves aren't always gauge-invariant, the average decoding accuracy remains invariant under a gauge transformation, which is a different kind of invariance than what we are looking for in the RSM <ref:2605.21324#pg1>.

Marcus: That distinction between decoding accuracy invariance and RSM dependence is crucial; it means even if the error rate doesn't change when you rotate the encoding, the similarity structure captured by the RSM can still shift because of that rotation <ref:2605.21324#pg1>.

Yuki: This implies that relying solely on decoding performance metrics might give us a false sense of comparison if we ignore the underlying geometric configuration dictated by stimulus symmetries <ref:2605.21324#pg0>.

Ines: Moving into the results, they use a toy model involving neurons tiling a one-dimensional ring, where the gauge variable phi defines a global orientation and shows how this angle alters the off-diagonal elements of the RSM <ref:2605.21324#pg1>.

Marcus: That visual demonstration is really helpful for grasping the concept; it shows that changing phi qualitatively changes those similarity measures, meaning the geometry is genuinely different, not just a rotation of the entire structure <ref:2605.21324#pg1>.

Yuki: It's fascinating because it means we can have multiple mathematically valid tilings of the same input manifold that are functionally identical but result in distinct RSMs depending on how you fix the lattice configuration <ref:2605.21324#pg1>.

Ines: They also extend this to learning scenarios, showing that stochastic gradient descent or energetic regularization can produce sparse, drifting codes over time, which subsequently causes the RSMs to drift as well <ref:2605.21324#pg0>.

Marcus: So this isn't just a static problem with trained models; it applies to how representations evolve during training, leading to representations whose similarity structure changes dynamically <ref:2605.21324#pg0>.

Yuki: That drift over time connects back to population genetics because genetic drift causes phenotypic variations that can lead to different functional outcomes, and here we see a similar mechanism happening in the learned representation space <ref:2605.21324#pg0>.

Ines: They also looked at structural parameters, finding that increasing the number of receptive fields tends to reduce variability in the RSM, although for small tuning widths, variability can still be non-negligible <ref:2605.21324#pg0>.

Paper summary: Marcus: That suggests a trade-off between having more features and getting a more stable similarity structure; maybe we need to consider that when building models for genomics data where feature selection is key, this stability is important <ref:2605.21324#pg0>.

Yuki: This study also generalized the analysis to higher-dimensional symmetric manifolds, like tiling the hypersphere S d-one with reflection symmetry, where the RSM takes a form like RSM = Id + R R-one Id + R-one <ref:2605.21324#pg2>.

Ines: And in that higher-dimensional case, they found elements obey "non-trivial equality relations," which suggests the dependence on gauge angles is much more complex than what we see in lower dimensions <ref:2605.21324#pg2>.

Marcus: That complexity makes it harder to simplify the interpretation of similarity metrics when dealing with high-dimensional data structures, especially when trying to account for batch effects or other noise <ref:2605.21324#pg0>.

Yuki: It reinforces the idea that as biological systems become more complex, the mathematical machinery we use to compare their states needs to be able to handle richer symmetry structures rather than assuming simpler relationships <ref:2605.21324#pg1>.

Ines: In realistic applications, like autoencoders trained on rotated image data, they found that while reconstruction loss didn't show a clear dependence on the gauge variable phi, the CKA similarity did drop significantly as the gauge difference increased <ref:2605.21324#pg0>.

Marcus: That CKA similarity drop is what really tells us something about how sensitive our comparison metric is to these symmetries in a way that reconstruction loss isn't <ref:2605.21324#pg0>.

Yuki: It confirms their point that standard general-purpose vision models might not render latent stimulus rotations invisible when compared using RSM-based metrics, which is an important consideration for understanding how these models process visual data across different contexts <ref:2605.21324#pg0>.

Ines: So to wrap up the core finding of "Stimulus symmetries can confound representational similarity analyses," it means that if you don't account for these stimulus symmetries, you risk comparing representations that are functionally equivalent but geometrically distinct <ref:2605.21324#pg0>.

Marcus: And the implication is that to reliably interpret functional significance in representations where data symmetries exist, we absolutely need to design a metric that is invariant to those specific symmetry transformations, which usually requires knowing something about the stimulus space itself <ref:2605.21324#pg0>.

Yuki: This motivates future work toward creating metrics that respect these inherent structural constraints when comparing different biological or artificial systems <ref:2605.21324#pg0>.

Conclusion: Ines: It means that if we compare two brain states that are doing the exact same job but are generated by inputs with a certain symmetry, our similarity score might be misleading because the underlying geometry is skewed. Marcus, thinking about cohort effects and batch variability in genomics, this suggests that if our data has hidden symmetries related to how samples were collected or processed, we might be comparing fundamentally different structures even if the biological signal is consistent. Yuki, from a population genetic viewpoint, this hints that subtle variations in environmental input across different populations could lead to functional equivalents that are structurally distinct when viewed through the lens of representation space.

Marcus: Exactly, Ines; it points out that batch effects aren't just noise in the numbers; they can be artifacts arising from these underlying stimulus symmetries not being accounted for in our statistical framework. Yuki, if we think about evolutionary history, this could mean that functional traits might appear highly conserved across species because they share a symmetry constraint on their underlying sensory inputs, but those constraints impose different geometric arrangements on the neural code itself.

Yuki: That makes sense; if the symmetry is inherent to the physical environment or developmental constraints, then those constraints dictate a specific geometry for any valid representation of that input, and if we don't map that symmetry onto our comparison metric, we miss genuine functional equivalence across different contexts. Ines, so what is the actual recovery here for computational biology?

Ines: We recover the idea that simply measuring similarity isn't enough; we need a metric that respects those symmetries to truly understand the function being performed by the representation. Marcus, if we look at this from a data scientist standpoint, it tells us we need to be much more careful about how we define distance and similarity when dealing with structured data like biological measurements where underlying constraints are often present but invisible. Yuki, what's the next step for researchers trying to build these robust tools?

Farhad Pashakhanloo, Jacob A. Zavatone-Veth

Harvard University

q-bio.NC, cs.LG

Submitted: 2026-05-20

Updated: 2026-10-02

Comments: 17+25 pages, 8+12 figures

Code: https://github.com/fpashakhanloo/stimulus-sym-rsa

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 83/100

The gist: Stimulus symmetries can confound representational similarity analyses because functionally equivalent representations related by stimulus symmetries can possess qualitatively different

Key concepts

Representational Similarity Matrix (RSM)
The RSM measures how similar different neural encodings are to each other using an inner product. The paper shows that if representations are related by stimulus symmetries, they might not be related by simple orthogonal transformations in the representation space, causing the resulting RSMs to differ even if the functions are equivalent.
Gauge Dependence
Gauge dependence occurs when a mathematical quantity, like an RSM, changes depending on how you choose to represent or transform the underlying data. The study formalizes this by showing that for general data symmetries, the RSM is not automatically invariant under those transformations unless the encoding itself follows specific orthogonal rules.
Functional Equivalence vs. Geometric Difference
The core finding is that stimulus symmetries can make two representations functionally equivalent (doing the same task) but geometrically different in representation space. This means they are not related by a simple rotation or orthogonal transformation, which is what RSM invariance usually requires for comparison.

Terminology

Summary

Stimulus symmetries can confound representational similarity analyses because functionally equivalent representations related by stimulus symmetries can possess qualitatively different representational geometries, leading to distinct Representational Similarity Matrices (RSMs).

The core finding

Stimulus symmetries render many representations functionally equivalent, but these different configurations can lead to different RSMs.

This work demonstrates that stimulus symmetries can yield functionally equivalent representations that are not, in general, related by an orthogonal transformation in representation space, and therefore have distinct RSMs. This highlights a possible mismatch between function-preserving stimulus symmetries and the orthogonal transformations to which RSMs are invariant.

Formalizing Gauge Dependence

The paper formalizes the gauge dependence of the RSM for general data symmetries. It introduces a setting where data lies on a latent space Z upon which a compact group G acts, and observations x depend on z and nuisance variables ξ. The action of the symmetry group G is defined as g · x = x(g · z, ξ). The RSM is defined by the Euclidean inner product of encodings: RSMh(x, x′) = h(x)⊤h(x′). For the RSM to be gauge-invariant, it must satisfy a condition: there must exist an orthogonal matrix O ∈ O(n), depending on g but not on x, such that (g · h)(x) = O(g)h(x).

Gauge Invariance of Decoding Accuracy

The paper shows that while the encodings themselves are not generally gauge-invariant, the decoding accuracy remains gauge-invariant. The average error incurred by encoding x with h and decoding with f, E[h, f], is shown to be invariant under a gauge transformation: E[g · h, g · f] = E[h, f]. This is proven by showing that the action of G on the decoder can be defined such that it preserves the average error.

Gauge Dependence of RSM

The crucial result is that RSM gauge invariance requires the encoding to be an orthogonal linear representation of G. The paper demonstrates this using a toy model where data lies on a one-dimensional ring encoded by neurons with weights defined by an offset angle φ (the gauge variable). The resulting RSM is gauge-dependent, as changing φ qualitatively alters the off-diagonal elements, showing that the global rotation of the tuning curves does not translate to an overall rotation in the representation space.

Learning and Drifting Representations

The study extends to learning scenarios. In a two-layer autoencoder trained under SGD, representations drift over time without selecting a particular solution. Furthermore, stochastic gradient descent or energetic regularization can generate sparse, drifting codes, leading in turn to drifting RSMs. This demonstrates that extrinsic stimulus symmetries can lead to drifting representations over learning, with RSMs that change over time.

Factors Influencing Variability

The paper investigates how structural parameters affect the gauge dependence of the RSM. It shows that increasing the number of receptive fields suppresses variability in the RSM, but for small tuning widths, variability remains non-negligible. Furthermore, penalizing the L1 norm of activations does not fix a gauge when data is uniformly distributed on a manifold; instead, it leads to a quantized set of preferred gauge angles.

Higher-Dimensional Symmetries

The analysis is generalized to higher-dimensional symmetric manifolds. For tiling representations of the hypersphere S(d-1) with reflection symmetry, the RSM takes the general form RSM = Id + R R−1 Id + R−1. The structure of this matrix reveals that for d > 2, elements obey non-trivial equality relations, indicating a more complex dependence on gauge angles than in lower dimensions.

Real Data and Pretrained Models

In realistic settings, such as autoencoders trained on rotated image data (Kuzushiji-MNIST), the gauge dependence of the RSM can be distinguished from other sources of variability. The analysis shows that while reconstruction loss shows no clear φ-dependence, the CKA similarity drops significantly as a function of the gauge difference, confirming that standard general-purpose vision models need not render latent stimulus rotations invisible to RSM-based model comparisons.

Conclusion and Future Directions

The paper concludes that to reliably interpret functional significance in representations with data symmetries, one must account for these symmetries. It suggests that comparing representations in a way invariant to data symmetry requires designing a metric invariant to the symmetry transformations, which generally necessitates knowledge of the stimulus space. The study motivates future attempts at seeking such metrics.

The gist

Stimulus symmetries can confound representational similarity analyses because functionally equivalent representations related by stimulus symmetries can possess qualitatively different representational geometries, leading to distinct Representational Similarity Matrices (RSMs).

**(Self-Correction/Review: The summary is structured as requested, starts with the required one-line summary, uses bold headers for 3-5 sections, quotes key phrases, and adheres strictly to the provided text. The length is appropriate.

Improvements for AI systems

Here are the specific improvements that can be made to AI systems based on this research, categorized by technical focus:

)1. Robust Representation Comparison for Biological/Symmetric Data:

The core improvement is developing methods to compare neural representations (e.g., from deep learning models or biological networks) that are related by stimulus symmetries (like rotation).

  • Specific Improvement: Instead of relying solely on standard metrics like Representational Similarity Matrices (RSMs), systems should incorporate symmetry-aware similarity metrics, such as the Gauge-Invariant Kernel Alignment (CKA) derived in Section 3.4, or methods that explicitly account for gauge transformations (Section 3.3).

  • System Capability: This allows AI researchers to determine if two functionally equivalent representations are truly related by a simple rotation (orthogonal transformation) or if they possess qualitatively different underlying geometries that standard RSMs might erroneously flag as distinct.

)2. Robustness Against Training Dynamics and Drift:

The paper demonstrates that representations can drift over time during training, even when task performance is stable, and that this drift is linked to stimulus symmetries.

  • Specific Improvement: Implement continual learning architectures with explicit mechanisms to track or compensate for gauge drift in latent spaces. This could involve regularization schemes (like L1 penalties) designed not just for sparsity but also to maintain gauge invariance during optimization, as explored in Section 5.2 and 5.1.

  • System Capability: AI systems trained on dynamic data streams (e.g., visual tracking, robotics) can maintain a more stable internal model of the world by understanding that representation drift is not always a sign of catastrophic forgetting but can be a consequence of latent symmetry manipulation during learning.

)3. Geometry-Aware Feature Extraction for High-Dimensional Data:

The work provides analytical tools for understanding the structure of representations in high-dimensional, symmetric spaces (like spheres or tori).

  • Specific Improvement: Utilize the mathematical framework derived in Section 4 (e.g., the general form of RSMs for higher dimensions, Equation 169) to constrain or guide representation learning models. This involves designing loss functions that penalize representations whose geometry deviates from a desired symmetry structure (e.g., enforcing specific structures on the Gram matrix).

  • System Capability: AI systems processing complex sensory data (images, sensor readings) can be engineered to learn features that respect known physical or geometric symmetries (like rotational invariance), leading to more compact, disentangled, and robust feature spaces compared to standard deep networks.

)4. Inferring Latent Variables from Gauge Information:

The paper shows that gauge information (the offset angle in the latent space) can be inferred from the resulting representation statistics (e.g., RF envelopes).

  • Specific Improvement: Integrate an inference module into neural network pipelines that estimates the latent gauge variable based on observable output statistics, rather than assuming a fixed global orientation. This is analogous to Section 7's method for inferring the phase of RF envelopes.

  • System Capability: AI systems can achieve greater interpretability by identifying not just what an object is, but also how its features are aligned relative to an external frame of reference, improving alignment in tasks like pose estimation or scene understanding.

)5. Developing Symmetrically Constrained Architectures:

The paper highlights the conflict between the constraints imposed by equivariant architectures (linear representations) and the flexibility required for natural networks (non-linear encodings).

  • Specific Improvement: Design hybrid neural architectures that explicitly incorporate learned gauge invariance during training, perhaps by using techniques from Section 3.3 where weights are transformed via an orthogonal matrix dependent on the group element, or by using symmetry-aware loss functions to enforce gauge invariance in the hidden layer activations (Section 5.1).

  • System Capability: This leads to the creation of neural networks that are inherently more robust and interpretable when processing data with known symmetries, bridging the gap between mathematically ideal equivariant models and empirically observed non-linear systems.

Abstract

What can representational similarity matrices (RSMs) tell us about a neural code? As the popularity of these summary statistics grows, so too does the need for a more complete characterization of their properties. Here, we show that symmetries in network inputs can confound RSM-based analyses. Stimulus symmetries render many representations functionally equivalent, but these different configurations can lead to different RSMs. These different RSMs reflect qualitatively different representational geometries, ranging from disentangled to maximally-mixed codes. We show that stochastic gradient descent or energetic regularization can generate sparse, drifting codes, leading in turn to drifting RSMs. Moreover, we demonstrate that these phenomena are present in networks trained to encode image data, where the symmetry is latent. Our results illustrate the challenges inherent in comparing nonlinear neural codes, when functionally-equivalent representations are not related by a simple rotation.

Sources

Related papers