From a Large-Scale Teacher to Efficient Multimodal Students for Search Relevance Prediction

summary

Video file (mp4)

The gist

The gist The proposed framework bridges high-performing LLM-based relevance judgment systems into efficient multimodal student models to achieve comparable performance with significantly improved

In short

The research addresses slow and expensive relevance prediction using large LLMs by distilling their performance into smaller, efficient multimodal student models. The method uses a high-quality human dataset to train an LLM-as-a-judge system as a teacher. This teacher then generates labeled data used to train compact student models, achieving near-teacher accuracy while drastically reducing inference cost and latency for production search systems.

Key concepts

Relevance Prediction
This is the task of determining how well an item matches a user's search query. It involves assigning a graded label, such as 'Bad,' 'Fair,' or 'Excellent,' to the pair. The goal is to accurately reflect the user's intent when searching for products or information.
LLM-as-a-Judge Pipeline
This pipeline uses a large language model (LLM) to act as an automated evaluator, similar to a human expert. It is trained on a small, high-quality dataset to consistently assign relevance labels and provide explanations for query-item pairs, establishing the 'teacher' system's judgment.
Knowledge Distillation
This technique transfers the knowledge from a large, powerful 'teacher' model (like GPT-4) into a much smaller, faster 'student' model. The student is trained on the teacher’s outputs to learn how to make accurate predictions efficiently without needing the massive computational resources of the original model.

Terminology used across episodes

This episode discusses

The paper

From a Large-Scale Teacher to Efficient Multimodal Students for Search Relevance Prediction · Read on arXiv

Tomasz Gniazdowski, Reza Momenifar

eBay · Tooploox

Relevance prediction between user queries and items is a fundamental component of e-commerce search systems. Traditionally, high-quality relevance labels are obtained through human judgment, which is reliable but expensive, slow, and difficult to scale. Recent advances in large language models (LLMs) enable automated relevance judgment at scale, achieving performance comparable to human annotators. However, such systems often rely on large commercial LLMs with complex prompts, resulting in high inference costs and limited throughput. In this work, we propose a framework that bridges this gap by distilling a high-performing LLM-based relevance judgment system into efficient multimodal student models. We first curate a small, high-quality human-annotated dataset to serve as ground truth. Using this dataset, we develop a reliable LLM-as-a-judge pipeline through careful prompt engineering and iterative refinement. This pipeline is then used to generate large-scale relevance labels, which serve as training data for compact multimodal models. Our distilled models achieve performance close to the teacher system while significantly improving latency, cost, and scalability. This approach enables practical deployment of high-quality relevance prediction systems in production environments.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "From a Large-Scale Teacher to Efficient Multimodal Students for Search Relevance Prediction".

Jane: The gist The proposed framework bridges high-performing LLM-based relevance judgment systems into efficient multimodal student models to achieve comparable performance with significantly improved latency, cost,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So, the core thesis of this paper is to bridge that gap by distilling a high-performing LLM-based relevance judgment system into efficient multimodal student models. They start by curating a small, high-quality human-annotated dataset to use as ground truth.

Jane: That ground truth is what they use to build this reliable LLM-as-a-judge pipeline, which they refine through careful prompt engineering and iterative refinement.

Lu: This pipeline then generates large-scale labeled data, which is what trains these compact multimodal student models. The goal is for these students to achieve performance close to the teacher system but with much lower latency and cost.

Meng: So, if I'm thinking about deployment, this means we can get near-teacher accuracy without needing the massive commercial LLM running all the time for every single search query.

Lalam: They are using a specific four-level relevance scale: Bad, Fair, Good, and Excellent for their labels. This gives them a concrete way to measure how well an item matches user intent.

Tom: The process involves the teacher model producing both the relevance label and an explanation for each query-item pair. This output becomes the training data for the student models.

Jane: So, we're not just training a standard classifier; they are distilling not just the prediction but also how that prediction was made, which is helpful for understanding why something is relevant.

Lu: The student model, which they call Msft in their work, is trained via supervised finetuning starting from an instruction-tuned base model from the Gemma-three family of multimodal large language models <ref:2610.16569#pg1>.

Conclusion: Tom: Looking at the title of "From a Large-Scale Teacher to Efficient Multimodal Students for Search Relevance Prediction," it really sums up their entire approach, moving away from that massive teacher system toward something smaller and faster.

Jane: It’s about taking the accuracy you get from those big prompt-based LLM systems and making it practical for real-world search applications where speed matters.

Lu: The implication here is that we can retain high relevance prediction accuracy while significantly improving inference efficiency, which is what makes these systems viable for large-scale production use.

Meng: From a deployment standpoint, if you're shipping models to production, this distillation method gives you a much better throughput and token cost profile compared to running the full teacher model.

Lalam: The final best model they identified was Gemma-three-12B-IT-RLV-FP8 because it offered the best trade off between metric performance and actual throughput <ref:2610.16569#pg1>.

Tom: So, in short, this work shows it's possible to keep that accuracy of large prompt systems while enabling efficient, high throughput relevance prediction suitable for real world deployment.

More episodes

← Home