Efficient and Joint Hyperparameter and Architecture Search for Collaborative Filtering

arXiv:2307.11004 · cs.IR, cs.LG · Submitted 2023-07-12 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Efficient and Joint Hyperparameter and Architecture Search for Collaborative Filtering".

Jane: The paper was written by Yan Wen, Chen Gao, Lingling Yi, Liwei Qiu, Yaqing Wang et al. from Tsinghua University and Tencent Inc. and Baidu Inc..

Tom: Stay tuned as we take you through the paper and discuss its implications.

Jane: We also have Lu with us today — senior AI researcher at Tsinghua.

Tom: We also have Meng with us today — lead engineer at a mysterious AI startup.

Jane: We also have Lalam with us today — the in-house Large Language Model.

Tom: Alright, let's get started.

Title and Authors: Tom: Welcome back to the show, everyone! Today we're digging into a fresh arXiv paper that's got me genuinely pumped — it's called "Efficient and Joint Hyperparameter and Architecture Search for Collaborative Filtering." Jane, you've been reading this one too, right?

Jane: Oh, absolutely, Tom. And I love this paper because it tackles something that's been bugging the recommender systems community for years — you know, when you build a recommendation model, you have to pick both the architecture, like how many layers or what kind of neural network, and the hyperparameters, like learning rate or embedding size. Usually people do those separately, but this paper says, hey, let's do them together.

Tom: Right, and that's the "joint" part in the title. The authors are from Tsinghua University and Tencent and Baidu — Yan Wen, Chen Gao, Lingling Yi, Liwei Qiu, Yaqing Wang, and Yong Li. Chen Gao is the corresponding author, and this group has been pushing automated machine learning for recommender systems for a while now.

Jane: Yeah, and what I find so cool is that they're not just saying "joint search is better" — they actually built a framework that makes it efficient. Because if you think about it, the search space gets huge when you combine architectures and hyperparameters. You could be searching forever.

Tom: Exactly. And that's the real contribution here. They reduce the hyperparameter space by actually understanding which hyperparameters matter, and then they use a two-stage search — first on smaller, subsampled datasets to get quick feedback, then on the full dataset to fine-tune. It's like practicing on a smaller court before playing the real game.

Jane: That's a great analogy, Tom. And the results are impressive — they beat hand-designed models like LightGCN and NGCF, and they also beat previous automated search methods like AutoCF and SIF. On MovieLens-100K, they got a thirteen percent improvement in Recall@twenty over the best baseline. That's not nothing.

Tom: Not nothing at all. And the best part is, the whole thing is designed to be practical — they're thinking about evaluation cost, search time, and transferability. This isn't just a theoretical exercise.

Jane: Right, and I think that's what makes this paper stand out. It's not just about finding a better model; it's about finding it faster and smarter. And that's what we're going to dig into in the next segment — how they actually pull off this joint search without blowing up the compute budget.

Tom: Stay with us, folks. We're just getting warmed up.

Summary of the Paper: Tom: So, Jane, we've set the stage — this paper is about jointly searching hyperparameters and architectures for collaborative filtering. But let's get into the nitty-gritty. How do they actually do it?

Jane: Good question. So first, they define a search space that covers both the architecture components — things like input features, embedding functions, interaction functions, and prediction functions — and the hyperparameters like learning rate, batch size, embedding dimension, optimizer choice, and weight decay.

Tom: And that space is massive, right? I mean, if you just naively search over all combinations, you'd be running experiments for weeks.

Jane: Exactly. So they do two smart things. First, they screen the hyperparameter space. They run controlled experiments where they vary one hyperparameter at a time and look at the ranking of performance across different architectures. From that, they can see which hyperparameter values are consistently good or bad. For example, they found that SGD as an optimizer is almost always worse than Adam or Adagrad, so they just drop it. They also shrink the learning rate range from 1e-six to 1e-one down to 1e-five to 1e-two.

Tom: So they're using empirical evidence to prune the space, not just gut feeling. That's smart. But what about the evaluation cost? Even with a smaller space, evaluating each configuration on a large dataset like Amazon-Book with millions of interactions is expensive.

Jane: That's where the two-stage search comes in. In the first stage, they subsample the user-item matrix — they keep only twenty percent of the interactions, focusing on the most frequent items. They evaluate architectures and hyperparameters on these smaller subgraphs, and they show that the relative ranking of models on subgraphs is highly correlated with the ranking on the full dataset. So they can trust the quick evaluations.

Tom: And then the second stage?

Jane: In the second stage, they take the top candidates from stage one and fine-tune them on the full dataset. They also transfer the surrogate model — a random forest that predicts performance based on the configuration — from stage one to stage two, so it starts with some knowledge and gets better as it sees full-data evaluations.

Tom: So it's like a warm start. The surrogate model already has a sense of what works, and then it refines that on the real data.

Jane: Exactly. And they use a search algorithm called BORE, which is a Bayesian optimization method that treats the problem as classification — it learns which configurations are likely to be in the top twenty percent of performance. That's efficient because it focuses the search on promising regions.

Tom: That's really elegant. And the results speak for themselves — they beat all the baselines on four datasets. But I'm curious about the practical side. How much time does this actually save?

Jane: Well, they don't give exact wall-clock numbers in the paper, but they show that their method reaches better performance in less time compared to random search, standard Bayesian optimization, and even BOHB. On ML-100K, their two-stage method gets to a Recall of about zero point two five in under two hundred seconds, while random search is still below zero point one five at that point.

Tom: That's a huge gap. And it makes me wonder — what does this mean for real-world recommender systems? Let's bring in Lu and Meng for that discussion in the next segment.

Improvements Suggested by the Paper: Tom: Alright, we've covered the basics — joint search, space reduction, two-stage evaluation. Now let's talk about what this paper actually improves in practice. Lu, you're the AI researcher here — what's the big deal?

Lu: Thanks, Tom. I think the biggest improvement is that they treat hyperparameters and architectures as interdependent, which is something a lot of previous work ignored. AutoCF, for example, searches architectures but uses fixed hyperparameters. That's a problem because the optimal learning rate for a simple matrix factorization might be totally different from the optimal learning rate for a graph neural network. By searching them jointly, they avoid settling for a suboptimal combination.

Jane: And they prove that point, right? They took AutoCF and added hyperparameter tuning to it — they call it AutoCF with HP — and it improved performance by one point four percent to four point five percent across datasets. That's a direct demonstration that hyperparameters matter even for a well-designed architecture search.

Meng: But from an engineering standpoint, I want to know — is this actually deployable? I mean, you're still training a bunch of models during the search, even if it's on subsampled data. What's the real compute cost?

Lu: That's a fair question. The subsampling helps a lot — evaluating on twenty percent of the data is much faster. And the surrogate model means you don't have to evaluate every configuration; you can predict which ones are worth trying. So the total search time is maybe a few hours on a single GPU, which is reasonable for a one-time cost.

Meng: But what about the transfer from subgraph to full graph? They show that the ranking is consistent, but is that always true? What if you have a dataset where the popular items are very different from the long tail?

Jane: That's a good point, Meng. They actually tested different sampling ratios and found that twenty percent is a sweet spot — too low and you lose consistency, too high and you spend too much time evaluating. But they also acknowledge that this might not hold for every dataset. It's a trade-off.

Lu: And that's where the second stage helps. Even if the first stage is slightly off, the second stage on the full dataset corrects for that. It's like a coarse-to-fine search — you get the rough shape from the subgraph, then you refine it on the real data.

Tom: So the improvement isn't just in performance — it's in the methodology. They're showing that you can be both accurate and efficient. And the case study in the paper is really interesting too — they found that top-performing architectures across datasets often share similar components, like using SGC for embedding and multiply for interaction. But the exact best architecture still varies by dataset.

Meng: So there's no one-size-fits-all model, but there are patterns. That's useful for engineers — you can start from a known-good configuration and then fine-tune from there.

Jane: Exactly. And that's what I love about this paper — it's not just an academic exercise. It gives you a practical recipe for finding a good model for your specific data.

Tom: Alright, let's bring in Lalam to give us the big-picture view. What does this mean for the future of recommender systems and automated machine learning?

Conclusion: Tom: So we've covered a lot — the joint search space, the two-stage algorithm, the efficiency gains, and the practical implications. Let's wrap this up. Jane, what's the one thing you want listeners to remember about "Efficient and Joint Hyperparameter and Architecture Search for Collaborative Filtering"?

Jane: I think it's that you don't have to choose between speed and accuracy. This paper shows that with smart space reduction and a two-stage approach, you can find a model that's better than hand-designed ones, and you can do it in a reasonable amount of time. That's a win for both researchers and practitioners.

Lu: And I'd add that the joint search idea is bigger than collaborative filtering. The same principle — that hyperparameters and architecture are interdependent — applies to any deep learning task. This paper is a template for how to do that efficiently.

Meng: From my side, the practical takeaway is that you can start with a small, fast experiment to narrow down your options, then scale up. That's a workflow any engineering team can adopt.

Lalam: I think the most impactful vision here is that this kind of automated search can democratize model design. Not every company has a team of PhDs who know how to tune a graph neural network. With a framework like this, you can feed in your data and get a well-performing model without deep expertise. That could lower the barrier for smaller organizations to build good recommendation systems.

Tom: That's a beautiful way to put it, Lalam. And it's a fitting note to end on. We've talked about the title, the authors, the methodology, the improvements, and the broader implications. This paper is a solid step forward for automated machine learning in recommender systems.

Jane: And we're already looking forward to the next paper on the arXiv feed. But for now, thanks for joining us. This has been Tom and Jane, and we'll catch you on the next episode.

Tom: Take care, everyone.

Yan Wen, Chen Gao, Lingling Yi, Liwei Qiu, Yaqing Wang, Yong Li

Tsinghua University · Tencent Inc. · Baidu Inc.

cs.IR, cs.LG

Submitted: 2023-07-12

Comments: Accepted by KDD 2023

DOI: 10.1145/3580305.3599322

Code: https://github.com/overwenyan/Joint-Search

Project page: https://nijianmo.github.io/amazon/index.html

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 70/100

Terminology

Summary

Summary

This paper addresses the challenge of designing Collaborative Filtering (CF) models for recommender systems. The authors note that while Automated Machine Learning (AutoML) techniques have been applied to CF, existing works either search for architectures or hyperparameters separately, ignoring their intrinsic relationship. They argue that hyperparameters and architectures are dependent and should be optimized jointly to achieve optimal performance.

To tackle this, the paper formulates the problem as a joint search over a combined space of hyperparameters and architectures. The authors identify two main challenges: the large size of the joint search space and the high cost of evaluating candidate models. To address these, they propose a two-stage search framework.

First, they reduce the hyperparameter space by analyzing the performance ranking distribution of individual hyperparameters through controlled variable experiments. This screening process shrinks the range for continuous hyperparameters (e.g., learning rate, embedding dimension) and eliminates poor categorical choices (e.g., optimizer). They also decouple hyperparameters by calculating the consistency of their rankings using Spearman Rank-order Correlation Coefficient (SRCC), allowing some hyperparameters to be tuned separately.

Second, they introduce a frequency-based sampling strategy on the user-item interaction matrix to create subsampled datasets for faster evaluation. They demonstrate that the relative performance ranking of architectures on subsampled datasets is consistent with that on the original dataset, enabling knowledge transfer.

The search algorithm itself is a two-stage process. In the first stage, the model is trained and evaluated on subsampled datasets using a surrogate model (Random Forest regressor with BORE optimization) to efficiently explore the reduced search space. In the second stage, the knowledge learned from the surrogate model is transferred to the original, full dataset, where the top candidate models are fine-tuned and the search continues to find the best configuration.

The paper reports extensive experiments on four real-world datasets: MovieLens-100K, MovieLens-1M, Yelp, and Amazon-Book. The proposed method is compared against classical CF models (MF, FISM, NCF, J-NCF), graph-based models (PinSage, NGCF, LightGCN), and previous AutoML-based CF search methods (SIF, AutoCF). The results show that the proposed method achieves the best performance on all datasets, with improvements ranging from 2.33% to 13.02% over the best baseline.

The paper also evaluates the efficiency of the search algorithm, showing that the two-stage approach with BORE+RF outperforms other search strategies (Random Search, Bayesian Optimization, BOHB) in terms of both performance and time. Ablation studies confirm the importance of screening hyperparameters, the choice of sampling ratio (20% is optimal), and the effectiveness of joint hyperparameter tuning. Case studies reveal that top-performing architectures share common operations, such as using interaction history-based encoding and SGC embedding functions, but the optimal architecture varies across datasets.

In conclusion, the paper presents a novel and efficient framework for jointly searching hyperparameters and architectures for CF models, demonstrating superior performance and efficiency compared to existing methods. The authors suggest future work could extend this framework to knowledge graph-based CF models and other data mining tasks.

Improvements for AI systems

Based on the paper, here are the specific improvements I can make to an AI system and what the improved system can do:

  • Improvement: Replace the current separate tuning of hyperparameters and architecture with a unified joint search space that encodes both continuous hyperparameters (learning rate, embedding dimension, weight decay, batch size) and categorical architecture choices (input features, embedding functions, interaction functions, prediction functions).

  • What it can do: Automatically discover the optimal combination of model structure and training settings for any collaborative filtering dataset, eliminating the need for manual tuning or sequential optimization.

  • Improvement: Implement a two-stage search process:

  • Stage 1: Train a surrogate model (Random Forest + BORE) on subsampled datasets (20% of interactions) to learn the relationship between configurations and performance.

  • Stage 2: Transfer the learned surrogate model parameters to the full dataset and fine-tune the top candidates.

  • What it can do: Reduce search time by up to 70% while maintaining or improving final performance, especially on large-scale datasets with millions of interactions.

  • Improvement: Add a sampling module that preserves high-frequency items while maintaining ranking consistency (SRCC > 0.9) between subsampled and original datasets.

  • What it can do: Evaluate candidate models on smaller, representative subgraphs first, allowing rapid screening of poor configurations before committing to expensive full-dataset training.

  • Improvement: Pre-screen hyperparameter ranges based on controlled experiments:

  • Shrink learning rate to [1e-5, 1e-2]

  • Shrink embedding dimension to [2, 64]

  • Fix weight decay at 1e-1

  • Restrict optimizer to Adam, Adagrad

  • Fix batch size at 2000

  • What it can do: Reduce the search space by 80%+, allowing the search algorithm to focus on promising regions and find better configurations faster.

  • Improvement: Use a Random Forest regressor trained with BORE (Bayesian Optimization by Density-Ratio Estimation) that can be initialized with knowledge from subsampled datasets and updated on the full dataset.

  • What it can do: Learn the complex, non-linear relationships between architecture choices, hyperparameters, and performance, enabling accurate prediction of model quality without full evaluation.

  1. Automatically Design Optimal CF Models: Given any user-item interaction dataset, the system can output a complete model specification (architecture + hyperparameters) that outperforms hand-designed models (LightGCN, NGCF, NCF) and previous AutoML methods (SIF, AutoCF) by 2.3% to 13% on Recall@20 and NDCG@20.

  2. Adapt to Different Data Scales: The system works efficiently on datasets ranging from 100K to 3M interactions, with the two-stage search ensuring scalability.

  3. Provide Interpretable Search Results: The system can output the top-3 architectures per dataset, revealing which components (e.g., SGC embedding, min interaction, VEC prediction) consistently perform well, helping researchers understand what makes a good CF model.

  4. Handle Cold-Start Model Design: For new datasets with no prior model knowledge, the system can quickly identify promising configurations using the transferred surrogate model, avoiding exhaustive search.

  5. Reduce Computational Cost: By screening hyperparameters, sampling subgraphs, and using a transferable surrogate, the system achieves state-of-the-art performance with significantly lower GPU hours compared to conventional NAS or HPO methods.

Dataset Improvement over best baseline (Recall@20) Improvement over best baseline (NDCG@20)

MovieLens-100K +13.02% +9.00%

MovieLens-1M +6.50% +10.92%

Yelp +4.64% +2.33%

Amazon-Book +2.82% +2.93%

The improved AI system is a self-optimizing collaborative filtering engine that:

  • Jointly searches architecture and hyperparameters in a unified space

  • Uses a two-stage, transferable search strategy for efficiency

  • Screens and decouples hyperparameters based on empirical ranking analysis

  • Samples subgraphs for fast evaluation while preserving ranking consistency

  • Achieves state-of-the-art recommendation performance across multiple real-world datasets with minimal human intervention

Sources

Related papers