Federate the Router: Learning Language Model Routers with Sparse and Decentralized Evaluations

summary

The gist

The gist Large language models (LLMs) are increasingly accessed as remotely hosted services by edge and enterprise clients that cannot run frontier models locally, making routing queries to models

In short

The paper introduces a federated learning framework to train language model routers using private, decentralized query-model evaluation data. Instead of centralizing sensitive data, clients collaborate to learn a shared routing policy. This method improves routing accuracy and cost trade-offs by leveraging diverse local distributions while maintaining client privacy.

Key concepts

Federated Learning (FL): A collaborative machine learning approach where multiple clients train a shared model without sharing their raw private data. In this context, clients share learned routing policies instead of their specific query data, allowing the system to learn from diverse, decentralized evaluation sets while respecting privacy constraints.
MLP-Router: A parametric router that learns global estimators to predict expected accuracy and cost for all models based on a query's embedding. It uses a shared neural network trunk and model-specific heads to output predictions for every model, creating a policy that maximizes utility.
K-Means-Router: A nonparametric router that estimates routing utilities by exploiting the spatial locality within the query embedding space. Clients train local K-means models, share their cluster centers with a server, which then computes global centers to guide routing decisions based on proximity.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 18 episodes
Federated Learning — explained in 17 episodes
Retrieval-Augmented Generation (RAG) — explained in 16 episodes
Interpretability — explained in 15 episodes
Quantum Fisher Information (QFI) — explained in 15 episodes

This episode discusses

The paper

Federate the Router: Learning Language Model Routers with Sparse and Decentralized Evaluations · Read on arXiv

Baris Askin, Shivam Patel, Anupam Nayak, Andrea Vigano, Jiin Woo, Gauri Joshi, Carlee Joe-Wong

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Federate the Router".

Jane: The gist Large language models (LLMs) are increasingly accessed as remotely hosted services by edge and enterprise clients that cannot run frontier models locally,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: To recap, this paper introduces the "Federate the Router: Learning Language Model Routers with Sparse and Decentralized Evaluations" framework, which aims to let clients build a shared routing policy using only their local offline query evaluation data. The core thesis is that centralizing this data for router training is not feasible because it’s fragmented and privacy-sensitive.

Jane: They claim their method enables clients to collaborate and transfer supervision across semantically similar requests by sharing the routing policies they locally train on private datasets. This allows them to train on those diverse, privacy-sensitive query distributions, plus it supports sparse or imbalanced coverage of models.

Lu: The paper tackles two types of routers: a parametric multilayer perceptron router and a nonparametric K-means router, both designed to work under heterogeneous client query distributions and non-uniform model coverage. This flexibility is key for real-world use cases where you don't have everything in one place.

Meng: How does this translate to something practical? They are essentially letting the AI learn how good a model is for a specific type of question without ever seeing all the questions from every other client.

Lalam: It’s like if everyone in the world had a piece of a puzzle, and they could learn how their piece fits with others without showing anyone their whole picture. The goal is that this shared policy helps them find models that balance response quality against inference cost, which is essential when you can't run frontier models locally.

Conclusion: Tom: So, looking at the whole "Federate the Router: Learning Language Model Routers with Sparse and Decentralized Evaluations," what we see is a practical foundation for training LLM routers using data that is private and fragmented across many different users. They show that federated routing improves the accuracy cost frontier over baselines on both global and client-level test distributions.

Jane: The authors validate that federated learning provides a lower suboptimality against local routing, specifically showing a bound of C' (D) s two NF (one/D) delta compared to the suboptimality bound of C' (D i) s two NF (one/D i) delta for a single client.

Lu: This means federated learning offers a lower suboptimality, specifically O(one/sqrt D) versus O(one/sqrt D i), because the federated router optimizes over the dataset D = sum i D i <ref:2601.22318#pg1>. That's a significant statistical improvement when you’re dealing with many clients.

Meng: For someone just listening to this, it means that instead of having one big, centralized router that might not fit any client well, you can have a system where each client contributes to making the overall routing decision better for everyone without compromising their private data.

Lalam: It offers a practical foundation for training LLM routers from privacy-sensitive and fragmented data. It moves us toward a more decentralized way of finding the right model for any given query, which is really useful when models vary so widely in capability and price.

More episodes

← Home