Towards a Theoretical Understanding of Two Tower Recommendation Models

This paper, Towards a Theoretical Understanding of Two Tower Recommendation Models, investigates the asymptotic behaviors of the two tower model applied in two-stage recommenders that entail a strong convergence to the optimal recommender system. While two-tower models are widely used in production-grade systems (such as Netflix, Pinterest, and Amazon) to embed high-dimensional features of both users and items into a low-dimensional space, the author notes that its theoretical behaviors remain comprehensively unexplored.

Model Formulation

The two-tower model employs two deep neural networks, known as towers, which function as encoders to embed high-dimensional features of both users and items into a low-dimensional space. The recommendation is based on the dot product between the feature vectors extracted from the two towers, f(x u) and (i), expressed as:

R(x u, i) = f(x u), (i)

The optimization of this model is typically conducted via a cost function that minimizes the squared error between predicted and observed ratings, often including a penalty term to avoid overfitting in the deep neural network.

Theoretical Contributions

The primary contribution of the work is establishing asymptotic characteristics of the two tower recommender model concerning its robust convergence towards an optimal recommender system. The research establishes several key theorems:

• Approximation Error (Theorem 4.1): The paper provides a measure of the approximation error, stating that the true model can be effectively approximated by R, provided that the underlying true functions f* and* in Equation 3 have sufficient smoothness. This result holds irrespective of the value of L [number of layers], indicating that the approximation error of the two tower model can converge to zero with any number of layers.

• Robust Convergence (Theorem 4.5): The paper establishes that the model converges to the true model at a rate explicitly determined by the values of beta, d u, and d i. Specifically, the convergence rate is bounded by O p(-2 beta over 2 beta+d ui 2), where represents the set of observed ratings. The author observes that the rate of convergence of the two tower model increases as the smoothness of the true model improves or the maximum intrinsic dimensions of user and item features decrease. This rate is noted to be faster than the majority of the existing theoretical results.

• Top-K Retrieval Guarantees (Theorem 4.6): The paper establishes a novel theoretical connection between MSE minimization and Top-K retrieval performance, proving that the Top-k retrieval error converges as: R K I over K times O p(-2 beta over 2 beta+d ui). This provides the first theoretical justification for using MSE as a surrogate objective in two-tower retrieval systems, confirming that as the estimator converges to R*, the probability of missing the relevant item in the retrieval set vanishes.

Experimental Validation

The theoretical claims are validated through extensive experiments on four real-world datasets [Yelp, MovieLens-1M, Amazon-Books, and CiteULike] showcasing that our theoretical convergence rates predict empirical performance scaling.

• Synthetic Experiments: By varying the intrinsic dimension (d ui) and Hölder smoothness (beta), the study observe[s] a distinct power-law relationship between the sample size and the MSE. The results confirm that the convergence bottleneck is strictly governed by the complexity of the underlying manifold (d ui), and that the empirical results align with our theoretical prediction that the convergence rate improves as beta to infinity.

• Real-World Performance: On the Yelp dataset, the proposed T2 Rec stands best in each scenario, outperforming baselines such as rSVD, KNN, Co-Ca, SVD++, NeuMF, LightGCN, and DCN-v2, with improvements in test errors ranging from 12.6% to 81.3%.

• Comparison with Industry Variants: When compared to the state-of-the-art two-tower architecture FIT, the paper finds that while FIT outperforms T2 Rec marginally due to better constant factors provided by its meta query module and lightweight similarity scorer (LSS), both models exhibit the same asymptotic convergence rate. This confirms that the established bounds capture the fundamental statistical limit of the two-tower architecture class.

1. Manifold-Adaptive Architecture Scaling

• The Improvement: Implement a dynamic architecture controller that estimates the intrinsic dimension (d ui) and Hölder smoothness (beta) of the user and item feature manifolds during the initial training phases.

• What the improved system can do: Instead of using a static number of layers and embedding sizes, the system will automatically scale the embedding dimensionality and network depth. If it detects a high d ui (complex manifold), it will expand the embedding space to prevent information loss; if it detects high beta (smooth manifold), it will simplify the architecture to accelerate convergence and prevent overfitting.

2. Intrinsic Dimension Regularization (IDR)

• The Improvement: Add a regularization term to the loss function that penalizes the intrinsic dimension of the latent embeddings produced by the two towers.

• What the improved system can do: By forcing the model to map high-dimensional features into a lower-dimensional manifold (minimizing d ui), the system directly optimizes the convergence rate. This allows the model to reach optimal recommendation accuracy with significantly fewer observed ratings than standard two-tower models.

3. Top-K Informed Loss Weighting

• The Improvement: Replace standard Mean Squared Error (MSE) with a weighted loss function that prioritizes samples based on their impact on the Top-K retrieval error bound (R K I over K times Error).

• What the improved system can do: The system will assign higher loss weights to boundary items—those that are highly relevant but currently sit just outside the Top-K retrieval set. This shifts the optimization focus from general rating prediction to the specific task of maximizing retrieval precision, effectively bridging the gap between MSE minimization and actual recommendation performance.

4. Hybrid Meta-Query Scoring

• The Improvement: Integrate a lightweight meta-query module and a lightweight similarity scorer (LSS) into the standard two-tower architecture to address the constant factor gap.

• What the improved system can do: While the asymptotic convergence rate remains the same, this addition reduces the constant factors in the error bound. The improved system will provide significantly higher recommendation accuracy and lower error rates in the early-to-mid stages of training and in production environments, providing the best of both worlds between theoretical convergence and empirical speed.

Production-grade recommender systems rely heavily on a large-scale corpus used by online media services, including Netflix, Pinterest, and Amazon. These systems enrich recommendations by learning users' and items' embeddings projected in a low-dimensional space with two tower models (two deep neural networks), which facilitate their embedding constructs to predict users' feedback associated with items. Despite its popularity for recommendations, its theoretical behaviors remain comprehensively unexplored. We study the asymptotic behaviors of the two tower model applied in two-stage recommenders that entail a strong convergence to the optimal recommender system. We establish certain theoretical properties and statistical assurance of the two tower recommender. In addition to asymptotic behaviors, we demonstrate that recommendation with two tower architecture attains faster convergence by relying on the intrinsic dimensions of the input features. Finally, we show numerically that the two tower recommender enables encapsulating the impacts of items' and users' attributes on ratings, resulting in better performance compared to existing methods conducted using synthetic and real-world data experiments.