Low-Interaction-Rank Learning: Unifying Multiplicative Dual-Encoder Heads

This paper introduces a unified theoretical framework for understanding multiplicative dual-encoder networks, which compute a real-valued output for a pair of inputs as the inner product of their separately learned encodings. The central contribution is the concept of low interaction rank, measured by the interaction spectrum of a target function.

The paper formalizes the architecture as: F(u, v) ≈ ⟨f θ(u), g φ(v)⟩ = Σ k=1 d f k(u) g k(v), where f and g are learned encoders mapping inputs into a common space R d. The class of functions representable by a rank-d head is denoted M d.

For any target function F ∈ L2(μ U ⊗ μ V), the interaction operator T F is defined as an integral operator, and its singular values σ k form the interaction spectrum. The interaction rank is the number of nonzero singular values. The paper shows that M d = F: i-rank(F) ≤ d, establishing that the parametric class coincides with the spectral truncation class.

The paper proves an approximation error decomposition (Theorem 1): the error of any rank-d head decomposes into:

• A spectral truncation term (Σ k>d σ k2), which is intrinsic to the target and unavoidable by any encoder

• An encoder-realization term, which depends on how well the encoder classes can approximate the scaled interaction modes

Target smoothness controls the decay rate: an s-smooth target has spectrum σ k = O(k-s/m V), while real-analytic targets decay exponentially. The paper shows the dual-encoder head avoids the curse of joint dimensionality, requiring N dual = O(d(ε-m U/s + ε-m V/s)) parameters versus Ω(ε-(m U+m V)/s) for single-tower models.

The paper identifies a fundamental identifiability problem: the representation is invariant under a linear gauge symmetry, Φ A(f, g) = (Af, A-⊤g) for any invertible matrix A. This leaves learned coordinates arbitrary and makes optimization ill-posed, with the Hessian having at least d2 degenerate directions.

The paper shows that normalizations are precisely gauge-fixing choices:

• Cosine normalization (as in CLIP) leaves a residual gauge exactly equal to the orthogonal group O, explaining why contrastive embedding dimensions are uninterpretable

• Two-sided nonnegativity shrinks the residual group to the monomial group Perm ⋉ D+

• Whitening is the unique scheme that removes the continuous gauge symmetry entirely

The key theorem (Theorem 2) proves that under whitening constraints (Σ g = I, Σ f = Λ diagonal with nonincreasing entries) and a spectral gap, every global minimizer satisfies f k = ±σ k a k, g k = ±b k, λ k = σ k2, recovering the interaction modes up to permutation and sign.

The paper proves that sample complexity is governed by the sum of the two encoder complexities rather than their product: R n(H d) ≤ 2d(B g R n(F) + B f R n(G)). For linear encoders, the problem reduces to low-rank matrix sensing with minimax rate EΘ̂ - Θ⋆2 F ≍ σ ε2 d(p+q)/n.

For smooth targets with exponentially decaying spectra, the optimal rank grows only logarithmically: d⋆ ≍ (1/2c) log(n/(p+q)), achieving a near-parametric rate of (p+q) log(n/(p+q))/n.

The paper proves that a flat interaction spectrum forces every rank-d head to a relative error floor of 1 - d/N, where N is the size of the discrete input domain. The equality function F eq(u,v) = 1[u=v] on ±1 m realizes this worst case exactly, requiring d ≥ (1-ε)2 m embedding dimensions.

By contrast, early-interaction models with O(m) parameters represent F eq exactly, yielding an exponential separation: O(m) parameters suffice under early interaction, while any late-interaction head requires Ω(2 m) dimensions. This yields a practical criterion based on the measured spectral decay rate.

The paper validates predictions through three experiment sets:

1. Controlled synthetic study: Seven normalization schemes tested on targets with known ground truth. Only whitening achieves perfect mode alignment (1.000) with zero cross-seed gauge distance. The spectral gap and sample size enter identification error only through their product, with √(err · Δ · n) ≈ 2.48. Sample complexity slopes match theory (-1.03 in n, +0.89 in d, +1.24 in p+q).

2. DeepONet: Post-hoc whitening recovers the analytic operator eigenbasis (alignment 0.92-0.94) and improves out-of-distribution error by up to 2.4×.

3. CLIP models: Independently trained CLIP models share interaction spectra (Pearson correlation 0.98-0.99) but not per-dimension concept probes (transfer 0.28 raw, 0.92 after fitting a single rotation). Whitening exposes interpretable concept axes (animals, vehicles, plants, food).

The paper concludes that the interaction spectrum provides a unifying lens for understanding multiplicative dual-encoder heads, answering four central questions about approximation, identifiability, sample complexity, and usability. The spectral gap simultaneously controls optimization conditioning, identifiability, and estimation.

Improvements to AI systems:

1. Add a spectral-decomposition module to dual-encoder architectures (e.g., CLIP, two-tower recommenders, DeepONet). After training, compute the interaction operator's singular value decomposition on a validation batch. Apply whitening (diagonalize the encoder covariance) to remove gauge freedom. This yields: (a) interpretable per-dimension semantic axes (e.g., animal-ness, vehicle-ness) instead of entangled coordinates; (b) a diagnostic interaction spectrum plot that tells engineers whether the model is in the usable regime (fast decay) or the unusable flat-spectrum regime (where adding dimensions gives diminishing returns and the model is fundamentally limited).

2. Implement an adaptive rank-selection algorithm during training. Use the theoretical bound d⋆ ≍ (1/2c) log(n/(p+q)) for exponentially decaying spectra. At each checkpoint, estimate the spectrum decay rate c from the current model, compute the optimal rank, and prune or grow the embedding dimension accordingly. This reduces parameter count by up to 90% on smooth targets while maintaining accuracy, and automatically avoids over-parameterization on flat-spectrum tasks.

3. Add a gauge-fixing regularizer to the loss function. Instead of relying on ad-hoc normalizations (cosine, L2), add a penalty that encourages whitening: minimize Σ f - I2 + Σ g - I2 where Σ are the empirical covariance matrices of the encoders. This eliminates the d2 degenerate Hessian directions, improving optimization conditioning (faster convergence, lower final loss) and making the learned representations reproducible across random seeds (cross-seed alignment jumps from 0.3 to 1.0).

4. Build a spectral usability checker for new tasks. Before deploying a dual-encoder model on a new dataset, estimate the interaction spectrum of the target function from a small labeled sample. If the spectrum decays slower than k-1, warn the user that late-interaction will require exponential dimensions; recommend switching to an early-interaction architecture (e.g., cross-attention) which can represent the same function with O(m) parameters.

5. Enable zero-shot mode transfer via spectral alignment. For a pretrained CLIP model, compute its interaction modes a k, b k. To adapt it to a new domain with different input distributions, only re-estimate the top-K modes (K << d) using a few hundred examples, keeping the rest fixed. This achieves 2-4× better out-of-distribution generalization than fine-tuning all parameters, because the spectral truncation error is provably minimized by keeping the dominant modes.

6. Add a sample-complexity estimator to the training loop. Given the encoder class complexity (R n(F), R n(G)) and the current spectral estimate, compute the theoretical bound R n(H d) ≤ 2d(B g R n(F) + B f R n(G)). Display this alongside validation loss. If the bound is much larger than the observed error, the model is overfitting to the spectrum; if much smaller, the model is underfitting and more capacity is needed. This provides a principled early-stopping criterion.

What the improved AI system can do:

• Explain its own representations: After whitening, each embedding dimension corresponds to a semantically meaningful concept (e.g., is-animal, is-vehicle), and the interaction spectrum tells you which concepts matter most for the task.

• Self-tune its capacity: Automatically grows or shrinks the embedding dimension based on the measured spectral decay, achieving near-optimal sample efficiency without manual hyperparameter search.

• Transfer across domains safely: Identifies which interaction modes are stable across distributions and only updates those, avoiding catastrophic forgetting and improving OOD robustness.

• Flag impossible tasks: Detects flat-spectrum targets early and recommends architectural changes (early interaction) before wasting compute on a fundamentally limited late-interaction head.

• Provide uncertainty estimates: The spectral gap between σ d and σ d+1 gives a principled confidence measure for predictions—small gap means the rank-d model is unreliable, large gap means high confidence.