The Geometry of Polynomial Group Convolutional Neural Networks

The paper rigorously analyzes the Jacobian structure of polynomial group convolutional neural networks, establishing key mathematical identities and proving a recursive relationship for their derivatives. This work is crucial for understanding the linear dependencies within these complex deep learning architectures, particularly by determining when the kernel of the Jacobian vanishes or simplifies significantly based on layer structure and convolution parameters.

Deriving Coefficients via Recursive Identities

The derivation begins by establishing fundamental relationships for complex coefficients involving multiindices, such as J[rL-2 g i]. A key identity is presented in equation:

J[rL-2 g i] = - sum h in G sigma r (theta')[rL-1 hg i] L(h) / (r sigma r-1 (theta')[(r − 1)rL-2 g i])

Further coefficients, such as those for the multiindex [g 1, (rL-1 - 1)g i], are derived by expanding the left side of a related equation. This process shows that these coefficients can be expressed linearly in L, which is essential for constructing subsequent linear systems.

Inductive Construction of Jacobian Components

The authors employ an inductive approach on r 1 to define the main components of the Jacobian. They define O m and D k,i using sums over group elements h in G. The core result is presented in equation:

J[r 1 g 1, (rL-2 - r 1)g i] = sum m=0 r 1 O m C i m, r 1

This structure allows the construction of a large matrix whose entries are defined by sigma r-1 (theta')[]. The subsequent step involves constructing a linear system using these components. For i not equal to 1, the coefficient equation corresponding to [rL-2 g 1, (r-1)rL-2 g i] yields:

r sum r 1=0 L-2 J[r 1 g 1, (L-2 - r 1)g i] D r i L-2 - r 1 + r O rL-2 D 0, i = 0

This equation is linear in L, allowing the determination of dependencies among the derivatives.

**Determining Dependencies in L **

By analyzing the linear system derived above, the authors prove that the submatrix omitting the column h=e has a symbolically nonvanishing determinant. This critical finding implies that the kernel is one–dimensional and spanned by (1, 0,, 0), leading to the conclusion that L(h) = 0 for all h not equal to e. This completes the proof of a crucial lemma regarding the structure of L.

Proof of Proposition 4.4 via Induction

The final section proves Proposition 4.4 by induction on the number of layers, L. The base case L=1 is straightforward: J theta = J theta 1 (x * theta 1) (1) = x * 1, which vanishes if and only if 1 = 0. For the inductive step, assuming the proposition holds for L-1 layers, the authors use Lemma B.1 to relate to J = J theta' ('). By convoluting with theta L-1, they derive a key equation:

J theta r-1 = - theta'* *

Since theta' is general and L not equal to 0, Lemma B.2 implies that = lambda e for some nonzero lambda. This forces the structure of the derivative vector to be:

= (1,, L-1, lambda theta L)

The final step shows that any in the kernel must be a linear combination of L-1 specific vectors, completing the inductive proof.

Improvements to AI Systems:

1. Development of Kernel-Constrained Regularization (KCR):

• We must move beyond generic regularization (like L2 or Dropout) and incorporate structural constraints derived from the Jacobian kernel, J theta. The paper proves that J theta is not arbitrary but has a highly constrained, low-dimensional structure involving specific linear combinations of weight derivatives (theta l) and convolutional tensor structures.

• Implementation: Develop a regularization term R KCR(theta) added to the loss function L(theta):

L total = L(theta) + lambda times Projection of grad theta L(theta) onto (J theta) squared

This forces the optimization process to find parameters theta that are maximally sensitive (non-redundant) with respect to the loss function, effectively minimizing the projection onto the null space directions.

2. Structured Model Redundancy Analysis and Pruning:

• The explicit characterization of J theta provides a deterministic method for identifying redundant parameters or pathways within a deep convolutional architecture (CNN). If a weight update theta lies in the kernel, it implies that the change has minimal impact on the network's output mapping (theta).

• Implementation: Create an algorithmic module that computes the basis vectors spanning J theta at runtime. This allows for Kernel-Guided Pruning, where weights or filters corresponding to directions in the kernel are flagged as candidates for removal or severe regularization, leading to model compression while guaranteeing minimal performance degradation relative to the full model's output manifold.

3. Derivation of Identifiability Metrics:

• The paper provides a rigorous mathematical framework for determining if a deep network is locally identifiable (i.e., if small changes in weights lead to predictable, non-zero changes in the output).

• Implementation: Develop an Identifiability Score (IS) for any given trained model theta. This score quantifies the distance of the weight space from the kernel manifold. A low IS signals high redundancy, potential overfitting to specific, uninformative directions, or structural ambiguity that requires architectural redesign (e.g., adding novel skip connections or restructuring convolution layers).

Capabilities of the Improved AI System:

1. Guaranteed Minimum Sensitivity Optimization:

• The system can train deep neural networks to converge not just on a minimum loss value, but on a geometrically optimal weight configuration that maximizes the sensitivity of the output manifold to small parameter perturbations. This leads to models with superior generalization bounds because they are trained away from degenerate regions defined by J theta.

2. Adaptive Model Compression and Knowledge Distillation:

• The system can perform Self-Correcting Pruning. Instead of simply pruning weights based on magnitude (L1/L2), it prunes weights based on their contribution to the kernel basis. This ensures that the compressed model retains maximal functional capacity by eliminating only redundant parameters, leading to significantly smaller models with provably preserved performance characteristics.

3. Diagnostic Tool for Model Architecture Review:

• The system acts as a Structural Integrity Checker for network design. Given a proposed architecture (number of layers L, convolution sizes, etc.), it can predict the expected dimensionality and structure of J theta. This allows researchers to proactively correct architectural deficiencies—for instance, detecting if an over-parameterized block is introducing unhelpful degrees of freedom that obscure the true signal pathway.

4. Theoretical Guarantee for Training Stability:

• By providing the IS metric, the system can warn developers when training stability is threatened by approaching a degenerate region of the weight space, thus mitigating catastrophic failure modes often encountered in highly over-parameterized models where optimization trajectories become trapped near null-space directions.

We study polynomial group convolutional neural networks (PGCNNs) for an arbitrary finite group G. In particular, we introduce a new mathematical framework for PGCNNs using the language of graded group algebras. This framework yields two natural parametrizations of the architecture, based on Hadamard and Kronecker products, related by a linear map. We compute the dimension of the associated neuromanifold, verifying that it depends only on the number of layers and the size of the group. We also describe the general fiber of the Kronecker parametrization up to the regular group action and rescaling, and conjecture the analogous description for the Hadamard parametrization. Our conjecture is supported by explicit computations for small groups and shallow networks.