The Routing Plateau: Understanding the Accuracy Limits of LLM Routers

Many current LLM routing methods converge to a narrow performance range far below the oracle router, indicating fundamental limits in their ability to handle query-specific routing decisions. This study evaluates 21 diverse routing methods across five benchmarks and reveals that this routing plateau is caused by a correctness-prediction bottleneck where routers learn coarse, global patterns instead of fine-grained, instance-level signals.

The Routing Plateau Phenomenon

The central finding is the existence of a routing plateau, where many different router designs, including kNN-style methods, achieve very similar accuracy ceilings that remain significantly below the oracle router. This phenomenon is characterized by three specific empirical observations: (1) Similar top-end accuracy, where the top 15 routers differ by only a small margin; (2) Strong kNN-style routers, which remain competitive across benchmarks; and (3) a persistent oracle gap, as all existing routers trail the oracle by 10–30 percentage points.

The Correctness-Prediction Bottleneck

The plateau stems from the realization that routers must learn to infer which models are likely to answer a given query correctly, but current methods often fail to reliably learn this signal from available data and query representations. Specifically, current routers tend to learn coarse, global patterns of model capability rather than fine-grained, instance-level differences in correctness. This occurs because the set of models that produce correct answers for many queries varies in ways not always aligned with overall model capability; higher-average-performing models are not consistently the most reliable on every instance. Consequently, their shared reliance on global-model-capability signals causes them to converge to a narrow accuracy ceiling far below that of the oracle router.

Analysis of Query Difficulty

The study stratifies solvable queries into hard and easy subsets based on whether the set of correct models varies significantly. A query is called hard if it requires selecting the only correct model, or if there are multiple potentially correct models but the global top-1 model is incorrect on that instance. The analysis shows that hard queries make up only 11.1%–35.4% of each benchmark, yet account for 69.9%–90.8% of the overall gap. This disparity confirms that routers share the failure of routing hard queries because they rely on coarse estimates rather than instance-specific correctness prediction needed for these difficult cases.

Directions for Breaking the Plateau

To move beyond this limit, the paper explores three key levers: (1) larger training datasets, (2) stronger query encoders, and (3) end-to-end fine-tuning. The authors constructed a new large-scale routing training dataset with 300k queries and 2.8 million query–model correctness labels. By scaling the training set, upgrading the query encoder from MODERNBERT-BASE to MODERNBERT-LARGE, and fine-tuning the router end-to-end, they obtained a combined accuracy gain of up to 2.13 pp, which closes approximately 14.6% of the oracle gap.

Synergy of Scaling and Fine-Tuning

The ablation study demonstrates that individual changes provide only modest gains, but the full combination yields the largest improvement. The dominant gain appears when data scaling is combined with fine-tuning: moving from 30k FT with MODERNBERT-BASE to 300k FT with the same encoder improves accuracy by 0.77 pp. Furthermore, Adding the larger encoder on top gives the best result, Scaled-FT, yielding a total gain of 1.24 pp over the baseline. This suggests that data scaling, encoder scaling, and task-specific fine-tuning are complementary levers for moving beyond the current frozen-encoder plateau.

Conclusion and Future Work

The proposed Scaled-FT narrows the gap to the oracle but does not close it, as this gap is expected due to the fundamental difficulty of solving a query-only prediction problem. The authors suggest that future progress requires "richer instance-specific evidence beyond static query embeddings, such as model-pool-aware objectives that compare models jointly (e.g., pairwise loss function), lightweight lookahead signals from partial generation or confidence estimates, and richer query representations. They conclude that the most actionable near-term safeguards involve per-subgroup evaluation, transparent disclosure of pool composition at deployment, and open release of routing benchmarks."

The gist: many current LLM routing methods converge to a narrow performance range far below the oracle router. This phenomenon is caused by a correctness-prediction bottleneck where routers learn coarse, global patterns instead of fine-grained, instance-level signals. The authors show that scaling training data, strengthening query encoders, and end-to-end fine-tuning can improve accuracy by up to 2.13 pp for today’s routers.

Here are specific, actionable improvements for AI systems derived from the findings of this paper, categorized by the bottleneck they address:

)1. Move Beyond Coarse Global Patterns to Fine-Grained Instance-Specific Routing (Addressing the Correctness Prediction Bottleneck):

The core finding is that current routers learn coarse, global patterns instead of fine-grained, instance-level correctness signals.

Current routers tend to learn coarse, global patterns of model capability rather than fine-grained routing signals. For many queries, however, the set of models that produce correct answers varies in ways that are not always aligned with overall model capability.

Specific improvements:

Implement a novel routing objective that explicitly penalizes reliance on top-performing models when the query is classified as hard (i.e., instance-specific routing required). This could involve training the router to maximize accuracy on hard queries while simultaneously minimizing the divergence between its selection distribution and a ground truth distribution that favors models correct for that specific query instance, rather than just maximizing average performance.

Develop a mechanism to inject explicit instance-specific correctness prediction signals into the router's loss function. This requires moving beyond simple cross-entropy loss on binary labels (Ym) to incorporate features or representations that capture the subtle differences in model behavior for specific query embeddings, effectively forcing the model to learn which model is correct for this specific input, not just which model is generally best.

Introduce a model-pool-aware objective during training. This objective should compare models jointly (e.g., using pairwise loss functions) rather than treating them as independent entities whose performance is aggregated into a single global score. This would help the router learn to exploit nuanced trade-offs between models that are only correct on specific subsets of queries, which is critical for hard queries where the highest average-performing model fails.

2. Enhance Query Representation Power (Addressing the Encoder Limitation):

The paper suggests that stronger query encoders can provide richer signals to overcome the plateau.

We further study how to move beyond the plateau and find that larger training datasets, stronger encoders, and end-to-end fine-tuning can further improve routing accuracy.

Specific improvements:

Systematically upgrade the query encoder from current baselines (e.g., MODERNBERT-BASE) to larger models (e.g., MODERNBERT-LARGE or similar high-capacity representations). This richer representation should capture more complex semantic and structural features of the query, providing the router with more discriminative information to distinguish between models that might otherwise appear similar based on coarse embeddings.

Explore multi-encoder fusion techniques where a single query is processed by multiple encoders (e.g., sentence-transformer and BERT). The resulting fused representation could be used as input to the router, capturing complementary signals that enhance the ability to identify fine-grained routing differences across models.

3. Leverage Data Scaling and End-to-End Fine-Tuning (Addressing Training Limitations):

The most significant gains (up to 2.13 pp) were achieved by scaling data, using larger encoders, and end-to-end fine-tuning.

By scaling the routing training set from 30k to 300k queries, upgrading the query encoder from MODERNBERT-BASE (∼110M parameters) to MODERNBERT-LARGE (∼340M parameters), and fine-tuning the router end-to-end, we obtain a combined accuracy gain of up to 2.13 pp.

Specific improvements:

Implement a robust data augmentation and scaling pipeline for routing datasets, targeting 300k+ query instances with high-quality model correctness labels. This large dataset is essential for providing the dense supervision needed to learn fine-grained signals and should be used as the primary training source for any new router architecture.

Adopt end-to-end fine-tuning (FT) for all novel routing methods, moving away from fixed prediction heads on frozen embeddings. End-to-end FT allows the router to adapt its selection strategy directly to the downstream task performance and query representations, maximizing the utility of the scaled data and stronger encoders.

4. Operational Deployment Safeguards (Addressing Real-World Risks):

The discussion in Section 6.4 points out risks associated with deployment:

At deployment, scaling this concentrates traffic on a few “winning” providers, marginalizes specialist models that excel on minority query distributions—non-English queries, niche domains, atypical formatting—exactly the hard regions where current routers already fail.

Specific improvements:

Integrate model-pool-aware objectives directly into the deployment and monitoring phase. This involves continuously auditing the router's performance not just on aggregate accuracy, but on metrics that track performance across different model subgroups (e.g., language, domain specialization). If a specific subgroup of queries consistently performs poorly under the current routing strategy, an adaptive mechanism should trigger a re-calibration or adjustment to the routing weights for that subgroup.

Develop lightweight generation-time signals (lookahead) to supplement static query embeddings. Instead of relying solely on pre-computed features, integrate minimal confidence estimates or partial generation feedback during the inference phase to allow the router to make more dynamic, real-time corrections when it detects an instance where its current coarse prediction is likely erroneous.

LLM routing has become a popular approach to improve the cost-quality trade-off of LLM services by adaptively selecting a model for each query. Recent work has explored a broad range of routing methods, including clustering-based routers, learned classifiers, pairwise ranking, and confidence-based approaches. Our extensive study of 21 routing methods across five benchmarks reveals a consistent phenomenon that we call the routing plateau (Fig. 1): many methods, including kNN, achieve very similar accuracy and converge to a narrow performance range that remains far below the oracle router. Our analysis supports a correctness-prediction bottleneck hypothesis: current routers primarily learn global-average model performance trends rather than fine-grained, query-specific routing signals. As a result, they collectively fail on queries that require instance-specific routing decisions. Moreover, to understand whether the plateau can be alleviated with a better training setup, we construct a 300K-query benchmark (Nine-by-300k). More data, larger encoders, and end-to-end fine-tuning improve eight routers by 1.24 pp on average, but leave the plateau largely intact. These findings suggest that further progress may require inputs beyond the query itself, such as partial output trajectories that reveal how models attempt the task.