Product Interaction: An Algebraic Formalism for Deep Learning Architectures

summary

The gist

The gist: Product interactions provide a principled way to generate and organize algebraic expressions by increasing interaction order, revealing that convolutional and equivariant networks arise as

In short

Product Interactions propose an algebraic method to unify modern neural network architectures by viewing layers as compositions of multiplication operators. This framework systematically generates networks by increasing the 'interaction order'—the complexity of how data elements multiply together—revealing that CNNs, attention, and Mamba correspond to specific interaction orders.

Key concepts

Product Interactions: An algebraic formalism where neural network layers are built by composing multiplication operators defined over various algebras. This allows researchers to systematically generate and organize complex network structures by increasing the order of interaction between data elements.
Interaction Order: The polynomial degree of the input variable in an expression, representing how many data elements are multiplied together in a layer's operation. Higher orders correspond to more complex architectures like attention or Mamba, while lower orders yield simpler structures like convolutions.
Symmetry Constraints: Principles that require compatibility between a transformation (like translation) and the algebraic product. These constraints restrict the possible forms of interactions, dictating how data transformations must align with the underlying mathematical structure of the network layers.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 19 episodes
Federated Learning — explained in 17 episodes
Retrieval-Augmented Generation (RAG) — explained in 17 episodes
Quantum Fisher Information (QFI) — explained in 16 episodes
Interpretability — explained in 15 episodes

This episode discusses

The paper

Product Interaction: An Algebraic Formalism for Deep Learning Architectures · Read on arXiv

Haonan Dong, Chun-Wun Cheng, Angelica I. Aviles-Rivero

Tsinghua University · University of Cambridge

Modern neural architectures are largely developed as distinct empirical constructions, despite recurring principles across convolutional, attention, state-space, and equivariant models. We introduce Product Interactions, an algebraic formalism that unifies these architectures under a common construction principle and turns their design into a systematic procedure. The core shift is representing data as algebraic elements and treating multiplication, rather than linear transformation alone, as a primitive architectural operation. Architectures are generated by composing multiplication operators, with input dependence organized by self-interaction order. Within this formalism, we derive convolutional and equivariant networks as symmetry-constrained first-order product interactions, recover Mamba from state-space models as an increase in self-interaction order from one to three, and establish attention as a third-order interaction from composing the same operator. Equivariance is incorporated by constraining the algebraic product via symmetry: translation equivariance yields convolution, while SO(3)-compatibility produces the Clebsch--Gordan product and tensor-field interactions. This perspective recasts architecture design as an algebraic construction problem: the choice of algebra defines admissible interactions, symmetry constrains their structure, and composition generates higher-order architectures. The framework is thus generative. Following these principles, we construct a simplified-algebra dynamical layer that improves both accuracy and throughput over Mamba on CIFAR-10. Experiments show that interaction order affects accuracy non-monotonically and symmetry constraints boost learnability. Product Interactions provides a mathematical methodology for analyzing existing architectures and systematically designing new ones.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "Product Interaction: An Algebraic Formalism for Deep Learning Architectures".

Jane: The gist: Product interactions provide a principled way to generate and organize algebraic expressions by increasing interaction order,

Tom: First, who's behind it and why it matters.

Title and authors: Tom: We’re moving on to segment two where we talk about who wrote this. The paper is called "Product Interaction: An Algebraic Formalism for Deep Learning Architectures" and it was written by Haonan Dong, Chun-Wun Cheng, and Angelica I. Aviles-Rivero.

Jane: So, before we get into the details of what they did with product interactions, can you tell us what this paper is actually trying to say in plain English? What's the main point they are making about these algebraic expressions?

Tom: They’re saying that deep learning architectures have an underlying algebraic description. They build on the idea that layers are just compositions of an operator defined over a suitable algebra, and this gives us a systematic way to generate and organize those expressions by increasing their interaction order.

Lu: The authors use feature algebra and an auxiliary structural algebra whose tensor product provides a unified representation of signals, which is pretty powerful for encoding different kinds of information together.

Meng: So they are essentially providing a universal language—a set of rules—for how these complex network structures can be built, regardless of the specific task. That sounds like it could simplify model design significantly if we can leverage that formalism.

Lalam: It suggests that the complexity we see in different architectures isn't just arbitrary design choice, but rather a result of systematically increasing the order of interaction they are allowed to have.

The paper's summary: Tom: Now let’s look at the actual summary of "Product Interaction: An Algebraic Formalism for Deep Learning Architectures." They detail how they formalize equivariance by requiring compatibility between a transformation and the algebraic product, which recovers convolutional networks (translation), harmonic networks (SO(two)), tensor field networks (SO(three)), and SE(three)-equivariant attention as symmetry-constrained product interactions <ref:2602.02573#pg1>.

Jane: So, this is where the paper connects the abstract algebra to real things we know, like translation equivariance in convolutions or rotation equivariance for three dee data <ref:2602.02573#pg1>. What does that connection actually unlock for us?

Lu: It means we can use representation theory to determine the structure constants of these algebras, and those constants encode all those pairwise interactions between the elements of our algebra.

Meng: If we can fix those structure constants according to specific objectives, like enforcing a certain symmetry, it’s a huge win for controlling how the model learns. It moves constraints from being just penalties to being built into the very structure of the network's operation.

Lalam: The paper also gives examples of algebras that facilitate auxiliary structural information, like positional information or higher-dimensional hidden state information, which means we can bake in things like sequence indices or spatial samples directly into the algebra.

The paper's improvements: Tom: Moving to the improvements section of "Product Interaction: An Algebraic Formalism for Deep Learning Architectures," they empirically demonstrate that interaction order, symmetry constraint, and algebraic structure significantly affect model performance. They suggest these three things are design principles for constructing product interactions.

Jane: So, what are these design principles in action? If we want a better model than the ones we have now, what does the paper suggest we should change about how we build it?

Lu: The paper suggests using a replacement principle: position Xone equals X makes the largest contribution to self-interaction. When accuracy drops significantly when you replace X by K, that tells you which parts of the interaction are truly important for learning.

Meng: That gives us a systematic way to analyze the contribution of each self-interaction term, which is really helpful when we’re trying to prune or simplify a model structure without losing too much performance.

Lalam: Also, incorporating symmetry constraints substantially facilitates learning unconstrained structure constants, which means we don't have to guess those parameters randomly; the symmetry guides us toward good settings.

Conclusion: Tom: We’re at the conclusion of "Product Interaction: An Algebraic Formalism for Deep Learning Architectures." To wrap up, they summarize that product interactions offer a principled approach to improving network effectiveness by leveraging the design of algebraic structures. It reduces various architectures to a single underlying operation: composing multiplication operator and then taking the product element as input.

Jane: So, what does this mean for the future of designing models? What's the big picture here?

Lu: It provides a principled way to construct interactions with any self-interaction order. This means we can systematically investigate how to effectively increase that self-interaction order in product interactions, which opens up new avenues for architectural design.

Meng: From an engineering standpoint, this framework gives us a systematic perspective for investigating alternative algebraic constructions. It’s not just another layer; it's a way to understand the underlying physics of how the computation flows.

Lalam: I think this formalism gives us a principled perspective for looking at different algebraic constructions, which is exactly what we needed to move past ad-hoc architectural design and toward something more systematic.

More episodes

← Home