Reducing Symmetry Increase in Equivariant Neural Networks

Equivariant Neural Networks (ENNs) have empowered numerous applications in scientific fields. Despite their remarkable capacity for representing geometric structures, ENNs suffer from degraded expressivity when processing symmetric inputs: the output representations are invariant to transformations that extend beyond the input’s symmetries. The mathematical essence of this phenomenon is that a symmetric input, after being processed by an equivariant map, experiences an increase in symmetry. While prior research has documented symmetry increase in specific cases, a rigorous understanding of its underlying causes and general reduction strategies remains lacking. In this paper, we provide a detailed and in-depth characterization of symmetry increase together with a principled framework for its reduction: (i) For any given feature space and input symmetry group, we prove that the increased symmetry admits an infimum determined by the structure of the feature space; (ii) Building on this foundation, we develop a computable algorithm to derive this infimum, and propose practical guidelines for feature design to prevent harmful symmetry increases. (iii) Under standard regularity assumptions, we demonstrate that for most equivariant maps, our guidelines effectively reduce symmetry increase. To complement our theoretical findings, we provide visualizations and experiments on both synthetic datasets and the real-world QM9 dataset. The results validate our theoretical predictions.

The paper's main contributions are:

• In § 3, we prove for any given feature space and input symmetry group, that the increased symmetry is bounded from below by a unique symmetry infimum, which is determined entirely by the algebraic structure of the feature space.

• In § 4, we develop a computable algorithm to derive this infimum by analyzing the orbit types. This provides practical guidelines for predicting and controlling potential symmetry increases, thereby preventing harmful symmetry increases in feature design.

• In § 5, we demonstrate that under regularity conditions, such as the manifold hypothesis for data, our method can fully reduce symmetry increase. Specifically, for most equivariant maps or for ENNs with sufficient approximation capabilities, the output symmetry will be precisely this predictable infimum, preventing orientational information loss.

• In § 6, we complement our theoretical findings with empirical evidence. We provide visualizations to illustrate the proposed concepts and present experimental results on both synthetic datasets and the real-world QM9 dataset, which validate our theoretical predictions and demonstrate the practical effectiveness of our framework.

The paper establishes that for any given feature space and input symmetry group, the increased symmetry is bounded from below by a unique symmetry infimum, determined entirely by the algebraic structure of the feature space (Theorem 3.1). This infimum is defined as the unique minimal orbit type among the points in the fixed-point subspace X H. A necessary condition for the existence of an isovariant map (one that strictly preserves symmetry) is that the orbit types of the input space are contained in those of the feature space (Theorem 3.2). When considering non-trivial kernels, the paper introduces an operator pY and defines isovariance relative to Y, with a corresponding necessary condition (Theorem 3.3).

The paper develops a computable algorithm to derive this infimum by analyzing orbit types, leveraging Michel's Criterion and proving its sufficiency for high-multiplicity representations (Proposition 4.2). This leads to practical guidelines for feature design: for orientation-dependent tasks, one should select feature components that contain the orbit type (pY(H)); for general tasks, one should avoid components where the symmetry infimum indicates a severe compression of the fixed-point subspace, particularly those that cause symmetry to increase to the full group.

The paper demonstrates that under regularity conditions (the manifold hypothesis), for most equivariant maps or for ENNs with sufficient approximation capabilities (such as TFN, which is proven to be C∞-dense in the space of smooth equivariant maps in Theorem 5.1), the output symmetry will be precisely this predictable infimum (Theorem 5.2). This ensures that any undesired increase in symmetry occurs only on a negligible portion of the data, and full relative isovariance can be achieved by increasing representation multiplicity.

The experimental results validate the theoretical predictions. Visualizations of representation spaces for k-fold structures show three types of degeneration: full degeneration (symmetry increases to O(3)), axial degeneration (symmetry increases to D∞h), and half degeneration (symmetry increases to D2kh). Experiments on symmetric graph discrimination tasks show a clear binary pattern in embedding difference norms, aligning precisely with theoretical predictions. Experiments on the QM9 dataset for molecular isotropic polarizability prediction show that for non-trivial feature components where molecular symmetry increases to O(3), the prediction loss is substantially higher, validating the design guidelines.

Improvements to AI systems:

1. Symmetry-Aware Feature Selection for Equivariant Neural Networks (ENNs):

• What it does: Automatically filters or designs feature spaces (e.g., irreducible representations in TFN) to avoid components that cause harmful symmetry increases (e.g., loss of orientational info).

• Specific capability: Given a task (e.g., molecular property prediction), the system computes the symmetry infimum for each candidate feature component and prunes those that force symmetry to inflate to the full group (e.g., O(3)). This reduces prediction error on orientation-dependent properties like polarizability or dipole moments.

2. Predictive Symmetry Auditing for Trained ENNs:

• What it does: After training, the system analyzes the learned equivariant map’s output symmetry on symmetric inputs, using the paper’s orbit-type algorithm to detect when and where symmetry increase occurs.

• Specific capability: It can flag data points or feature channels where the model is invariant to transformations it should not be (e.g., rotations of a chiral molecule), enabling targeted retraining or data augmentation to fix these blind spots.

3. Optimal Representation Multiplicity Selection:

• What it does: Uses Theorem 5.2 to determine the minimum number of copies (multiplicity) of each feature component needed to achieve full relative isovariance (i.e., no symmetry increase beyond the input’s inherent symmetry).

• Specific capability: For a given ENN architecture and dataset, the system automatically sets the multiplicity of each representation (e.g., l=1,2,3 in TFN) to guarantee that the output symmetry matches the input symmetry exactly, preventing information loss without over-parameterizing.

4. Symmetry-Informed Data Augmentation for General Equivariant Models:

• What it does: Uses the paper’s characterization of symmetry increase to generate adversarial symmetric inputs that cause the model to collapse to higher symmetry, then augments training with those inputs to force the model to learn non-collapsing representations.

• Specific capability: Improves robustness of any equivariant model (e.g., graph neural networks, point cloud processors) by ensuring it distinguishes between inputs that differ only by symmetries not present in the data, reducing false positives in classification or regression.

5. Isovariant Regularization Term:

• What it does: Adds a penalty to the loss function that measures the deviation from the predicted symmetry infimum (computed via the paper’s algorithm) for each feature component.

• Specific capability: During training, the system penalizes any output that exhibits symmetry higher than the infimum for the given input, explicitly discouraging symmetry increase. This yields models that preserve more fine-grained geometric information, improving performance on tasks like 3D object pose estimation or molecular conformation generation.

6. Automatic Detection of Degenerate Feature Spaces in Pre-trained ENNs:

• What it does: For an existing ENN (e.g., a pre-trained model on QM9), the system runs the orbit-type analysis on its feature layers to identify which channels have severe symmetry compression (e.g., full degeneration to O(3)).

• Specific capability: It can then either fine-tune only those channels with higher multiplicity or replace them with better-designed components, improving downstream task accuracy without retraining the entire network.

7. Symmetry-Aware Loss Weighting for Multi-Task Learning:

• What it does: For tasks with different symmetry requirements (e.g., predicting scalar energy vs. vector dipole), the system assigns higher loss weights to feature components that are prone to symmetry increase, based on the paper’s infimum calculation.

• Specific capability: In multi-task molecular prediction, the system ensures that orientation-sensitive tasks receive stronger gradient signals from non-degenerate feature channels, preventing the model from sacrificing vector accuracy for scalar accuracy.

8. Theoretical Guarantee for Equivariant Universal Approximators:

• What it does: Uses Theorem 5.1 (TFN is C∞-dense) and Theorem 5.2 to certify that, for a given architecture, the model can achieve the exact symmetry infimum on all but a measure-zero set of inputs, provided the feature space is designed per the paper’s guidelines.

• Specific capability: Enables the system to provide a formal certificate of “no harmful symmetry increase” for a trained ENN on a given dataset, which is critical for safety-critical applications like drug discovery or materials design where orientational information is physically meaningful.

What the improved AI system can do overall:

• Design equivariant architectures that provably avoid symmetry-induced information loss.

• Diagnose and fix symmetry collapse in existing models.

• Guarantee that predictions respect the true symmetries of the input, not accidental higher symmetries.

• Improve accuracy on scientific tasks (e.g., QM9, molecular dynamics) where orientation matters, by up to the margin shown in the paper’s experiments.

• Provide a computable, theory-backed tool for feature engineering in any equivariant learning pipeline.