ModeSwitch-LLM: A Lightweight Phase-Aware Controller for Cross-Mode LLM Inference on a Single GPU

arXiv:2605.23057 · cs.LG, cs.CL, cs.PF · Submitted 2026-08-19 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "ModeSwitch-LLM: A Lightweight Phase-Aware Controller for Cross-Mode LLM Inference on a Single GPU".

Jane: The paper was written by Aman Sunesh, Ali Alshehhi and Hivansh Dhakne from NYU Abu Dhabi and Courant Institute of Mathematical Sciences, New York University and Department of Computer Engineering, New York University.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Summary: Tom: Now that we understand the mechanism, let’s look deeper into the summary section, which clearly explains how ModeSwitch-LLM works by looking at its core components. It's important to see exactly what the controller is doing under the hood.

Jane: The core idea here is that the system acts as a sophisticated dispatcher, routing each incoming request to one of several "fixed inference modes" using a set of deterministic rules, which means it doesn't rely on probabilistic guessing.

Lu: And what makes this approach so innovative is that it uses cheap workload-level features—things like prompt length and shared-prefix structure—to make the routing decision before the generation even begins.

Meng: When we talk about "cheap" features, I need to know if these require complex preprocessing or heavy computation, because for a low latency serving environment, any overhead is absolutely detrimental.

Lalam: No, Meng; it means we are talking about basic metadata—simple checks on the prompt length and the shared-prefix structure—which can be extracted almost instantly.

Tom: That's right; the summary emphasizes that this allows us to predict the needs of an AI query using fundamental data points rather than having to wait for a complex, resource-heavy analysis during inference itself.

Jane: This routing intelligence is applied directly to existing optimization strategies, meaning we don't have to retrain or change the underlying model architecture at all.

Lu: I found it particularly interesting that they demonstrated combining modes in hybrid configurations, for instance, pairing GPTQ quantization with prefix caching for tasks involving shared-prefix structures.

Meng: This combination is very pragmatic because the required features are available right at request time with negligible extraction cost, which is essential for maintaining low latency.

Lalam: It allows us to provide a tailored AI experience, giving a quick response for a short chat versus running an optimized process for a long-form generation.

Tom: It really highlights that the intelligence is in the dispatching mechanism, not in developing an entirely new model or architecture.

Jane: We see specialized modes being used based on simple characteristics of a request, like if it's a short Q andA versus a long document summary task requiring more computational power.

Lu: This concept of dynamic pairing between different optimization methods is what makes the system so flexible and powerful for real-world deployment.

Meng: It’s an elegant solution that directly addresses the tension between maximizing efficiency and ensuring high usability for diverse user base needs.

Lalam: So, they are providing a single point of entry that optimizes the entire pipeline based on minimal initial inspection of the query parameters, which is very efficient.

Tom: This understanding of how routing works is critical because it leads us directly into the measurable outcomes: what improvements did this system actually achieve?

Improvements: Tom: The results section really shows the payoff of this concept, and I think it’s impressive what they achieved with ModeSwitch-LLM, especially when looking at those key performance metrics.

Jane: On synthetic workloads, they achieved a significant two point one zero times mean latency speedup compared to standard FP16 serving, which is an enormous gain for any real user experience where milliseconds matter.

Lu: But it's not just about speed; the energy impact is equally impressive—fifty-one point seven percent lower energy per token—which means we are seeing efficiency gains that have massive environmental implications.

Meng: For a startup or a large data center, the practical implication of those numbers translates directly into reduced infrastructure costs while handling wildly diverse workloads simultaneously.

Lalam: This suggests that we can provide faster, greener AI services without having to compromise the quality of the output, which is absolutely essential for widespread societal adoption.

Tom: Beyond just speed and efficiency metrics, they also show how well this system maintains benchmark accuracy even with all these optimizations running concurrently.

Jane: Even with all those optimizations happening—quantization, caching, etc.—the accuracy stays very close to the standard FP16 baseline, showing only a mean delta of about +zero point one seven percentage points.

Lu: That small deviation is quite telling; it demonstrates that the optimization isn't just cutting corners or making rough approximations but finding pathways that maintain fidelity to the original model behavior.

Meng: It practically proves that these different inference modes are not fundamentally incompatible with each other, which makes the whole system highly scalable in terms of hardware utilization.

Lalam: This capability means we can finally move closer to having a highly optimized AI service where we don't need massive over-provisioning just to handle unpredictable peak usage demands.

Tom: So, the improvements section really solidified that this controller is not just a theoretical concept, but a highly viable optimization layer for deployment.

Jane: We’ve covered speed and efficiency, but there was one key comparison they made that deserves another look: how does this compare to complex learning methods?

Lu: That brings us to the ultimate question of whether simple rules are truly enough, or if we need a full machine learning router that tries to predict the best path for every query.

Meng: It’s interesting because it seems like a robust rule-based system is performing almost as well as a complex learning model, which simplifies things greatly for real deployment.

Lalam: This capability allows us to build AI that is both high-performance and reliable, ensuring consistency whether the user is doing simple tasks or complex ones.

Tom: The data shows that the controller isn't just a neat trick; it’s a highly functional way to make inference more efficient.

Conclusion: Tom: Looking at the overall findings from ModeSwitch-LLM: A Lightweight Phase-Aware Controller for Cross-Mode LLM Inference on a Single GPU, what's the final word on the impact of this approach?

Jane: It’s clear that modern AI services don't require massive hardware investment if we can manage traffic intelligently like this, allowing for much more efficient resource usage across different workloads.

Lu: I think that’s where the real excitement lies; how does this hint at scaling up these intelligent routing concepts across distributed environments in the future?

Meng: The engineering takeaway is that a simple rule-based system delivers the most immediate and measurable ROI for deployment, proving we don't need complex machine learning models to achieve best results.

Lalam: This allows us to build AI that is not just powerful, but also responsible and efficient for every single interaction with society, which brings a sense of shared responsibility.

Tom: That’s exactly the balance they achieved—efficiency without sacrificing quality, which is a major hurdle in many other systems.

Jane: And Lu’s point about scaling really depends on the idea that this system is deterministic; it doesn't rely on probabilistic guessing from a complex classifier.

Lu: Exactly, and I think that simplicity is what makes this such a clean blueprint for future architecture design across many different types of AI workloads.

Meng: We can finally implement these rulesets and optimize the features with confidence, which is much more practical than training an unreliable machine learning classifier that often has high overhead.

Lalam: It allows us to deliver a high-quality experience, whether the user is doing a quick chat or a long document analysis.

Tom: I’m just hoping that this paves the way for even more refined systems down the road, leveraging these foundational principles of intelligent routing.

Conclusion: Tom: To wrap up our look at this research, we’ve seen how effective ModeSwitch-LLM is at optimizing inference on a single GPU by intelligently routing requests based on simple workload features.

Jane: It's clear that modern AI services don't require massive hardware investment if we can manage traffic intelligently like this, reducing the pressure to increase server size.

Lu: I think that’s where the real excitement lies; how does this hint at scaling up these intelligent routing concepts across distributed environments in the future, leveraging those same core principles?

Meng: The engineering perspective shows us that a simple rule-based system delivers highly measurable ROI for deployment, proving we don't need complex machine learning models to achieve best results.

Lalam: This allows us to build AI that is both powerful and responsible, ensuring every interaction with society benefits from efficiency and reliability.

Tom: That’s exactly the balance they achieved—efficiency without sacrificing quality, which is a major hurdle in other papers we've discussed today.

Jane: And Lu’s point about scaling really depends on the idea that this system is deterministic; it doesn't rely on probabilistic guessing from a complex classifier.

Lu: Exactly, and I think that simplicity is what makes this such a clean blueprint for future architecture design across many different types of AI workloads.

Meng: We can finally implement these rulesets and optimize the features with confidence, which is much more practical than training an unreliable machine learning classifier that often has high overhead.

Lalam: It allows us to deliver a high-quality experience, whether the user is doing a quick chat or a long document analysis.

Tom: We hope this paves the way for even more refined systems down the road, building upon what ModeSwitch-LLM: A Lightweight Phase-Aware Controller for Cross-Mode LLM Inference on a Single GPU offers us today.

Aman Sunesh, Ali Alshehhi, Hivansh Dhakne

NYU Abu Dhabi · Courant Institute of Mathematical Sciences, New York University · Department of Computer Engineering, New York University

cs.LG, cs.CL, cs.PF

Submitted: 2026-08-19

Updated: 2026-08-21

Comments: 5 pages, 2 figures. Accepted at the 2nd International Workshop on Low Carbon Computing (LOCO 2026), Lancaster University, United Kingdom, 10-11 September 2026. Part of the LOCO 2026 proceedings, arXiv:2608.02072

Code: https://github.com/ModeSwitch-LLM/ModeSwitch-LLM

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 89/100

The gist: ModeSwitch-LLM is a lightweight request-boundary controller designed to address the challenge that large language model (LLM) inference workloads are heterogeneous, yet traditional serving systems

Key concepts

Phase-Aware Controller
This system acts as a sophisticated dispatcher, routing incoming AI requests to specific 'fixed inference modes' using deterministic rules. This intelligent dispatching mechanism allows the system to tailor the AI experience based on simple characteristics of a request.
Workload-Level Features
These are basic metadata used for routing decisions, such as checking prompt length or shared-prefix structure. They are considered 'cheap' because they can be extracted almost instantly, avoiding the need for complex, resource-heavy analysis during the inference process.
Cross-Mode LLM Inference
The system optimizes the entire AI pipeline by dynamically pairing different optimization methods, such as GPTQ quantization and prefix caching. This allows for a tailored experience, providing quick responses for short chats or optimized processing for long-form generation.

Terminology

Summary

ModeSwitch-LLM is a lightweight request-boundary controller designed to address the challenge that large language model (LLM) inference workloads are heterogeneous, yet traditional serving systems often rely on a single static configuration that is not optimal across all workload types. The core problem identified is that a configuration that is efficient for one workload can be inefficient for another.

Methodology and System Setup

The system uses Meta-Llama-3.1-8B-Instruct served on a single NVIDIA A100 40GB GPU via vLLM. The goal of the controller is to improve efficiency by selecting among fixed inference modes based on cheap workload-level features, rather than modifying the model architecture or retraining it.

The evaluation was conducted using two categories of workloads:

  1. Synthetic deployment-style workloads: These tests cover varying combinations of prompt length and expected output length (e.g., short-prompt/long-output, long-prompt/short-output), representing different stress tests for latency, decoding, and prefill computation.

  2. Automatic benchmark workloads: These include standard benchmarks such as MMLU-Pro, GSM8K, TruthfulQA, GPQA, and MLU to serve as a quality gate.

The ten candidate inference modes evaluated included the FP16 baseline alongside optimized options such INT8 quantization, GPTQ 4-bit quantization (Frantar et al. [2022]), AWQ 4-bit quantization (Lin et al. [2023]), speculative decoding, prefix caching, chunked prefill, continuous batching, and various hybrid configurations.

The Controller Design

ModeSwitch-LLM implements a request-boundary controller that operates in three steps: feature extraction, classification, and routing.

  1. Feature Extraction: The controller extracts six features at the time of the request: prompt token count, expected output token count, shared-prefix status, memory-pressure status, batch-pressure level, and workload tag.

  2. Classification: A lightweight rule-based classifier estimates the request type (e.g., batched or shared-prefix) based on these features. This approach was chosen for its interpretability and negligible routing overhead, rather than a learned classifier.

  3. Routing Policy: The the controller maps requests to one of five candidate modes (e.g., GPTQ 4-bit, speculative decoding, etc.). The specific priority policy dictates that Batched requests are routed to INT8 plus continuous batching and Shared-prefix chat requests are routed to GPTQ plus prefix caching.

The measured CPU routing overhead for this system is approximately 0.0096 ms per request.

Experimental Findings

The research found that no single inference mode is uniformly optimal across all workload shapes, motivating the use of a request-level router.

  • Fixed-Mode Screening: Initial benchmarking showed that different optimizations have distinct strengths; for instance, GPTQ 4-bit gives the strongest latency and energy improvements on many synthetic workloads, while prefix caching is useful mainly when repeated context is present.

  • Quality vs. Efficiency: The results demonstrated that efficiency gains do not always correlate with quality preservation. For example, while certain modes provide strong latency and energy benefits, their accuracy can drop significantly on specific benchmarks.

  • Online Controller Performance: When applied to deployment-style synthetic workloads, the controller achieves substantial efficiency gains: the online controller achieves a 2.10× mean latency speedup and a 0.48× mean energy ratio, corresponding to 51.7% lower energy per token. On automatic benchmark workloads, it maintains accuracy close to FP16 with a mean delta of +0.17 percentage points.

Comparison to Learned Routers

The study also compared the rule-based controller against trained routing policies (e.g., decision-tree and random-forest classifiers) designed to imitate a constraint-aware oracle. The learned routers were found not to clearly outperform the hand-written rule controller because they add routing overhead and tend to make more constraint-violating choices.

Conclusion

ModeSwitch-LLM successfully demonstrated that simple, phase-aware, and workload-aware rules can recover substantial efficiency in single-GPU LLM serving. The system achieves a 2.10× mean latency speedup and a 0.48× mean energy ratio on synthetic workloads while maintaining benchmark accuracy close to the FP16 baseline. The paper concludes that simple request-aware routing is already a strong practical baseline for efficient single-GPU LLM serving, without requiring model retraining or changing the existing architecture.

Improvements for AI systems

The core improvement is replacing a monolithic, static serving configuration with a dynamic, request-aware orchestration layer. This requires implementing a dedicated Pre-Inference Gate Controller within the LLM serving pipeline (e.g., in vLLM or custom Triton/Torch deployment).

  1. Deployment of the Workload Profiling Engine:

  • Establish a dedicated module responsible for real-time telemetry extraction upon request arrival, before any inference computation begins. This engine must gather six specific low-cost features:

  • L prompt (Prompt Token Count)

  • L output (Expected Output Token Count)

  • SharedPrefixFlag (Boolean, based on token overlap with cached context)

  • MemPressureStatus (Indicator of current GPU memory saturation relative to total allocated capacity).

  • BatchPressureLevel (Current queue depth/pressure on the serving queue).

  • WorkloadTag (Categorization based on benchmark family or synthetic profile).

  1. Dynamic Mode Selection Logic: Implement a fixed, priority-ordered routing policy that maps the extracted feature vector to one of ten defined modes. This is superior to using a complex learned classifier for performance-critical serving environments:

  • Optimal Routing Flow:

  • If BatchPressureLevel is high to INT8 + Continuous Batching. (Maximizes throughput).

  • If SharedPrefixFlag is true to GPTQ + Prefix Caching. (Maximizes reuse/latency reduction).

  • If MemPressureStatus is critical to GPTQ 4-bit. (Mitigates memory bandwidth issues).

  • If L prompt and L output are large to Speculative Decoding. (Accelerates decoding phase).

  • If the request profile matches a known benchmark/long-context stress test to INT8 Quantization (Ensures quality preservation).

*Default/Fallback: FP16 (Guaranteed baseline performance).

The system must be configured to leverage a suite of distinct, complementary fixed inference modes, rather than attempting to force one single best mode. These modes act as the operational levers controlled by the router:

  • Low-Precision Optimization (INT8/GPTQ 4-bit): Deploy highly optimized kernels for these modes, ensuring they are ready for immediate activation based on workload needs.

  • Context Reuse Mechanism (Prefix Caching): Implement a robust, high-speed prefix caching layer that is selectively activated by when the router selects the Shared-Prefix mode.

  • Batch/Throughput Optimization: Enable continuous batching capabilities in the serving runtime, specifically targeting its activation when queue pressure is detected.

  • Resource Monitoring & Quality Gate: Establish a continuous monitoring loop for both efficiency metrics (GPU power consumption via NVML) and quality metrics (ROUGE-L/Accuracy).

By implementing this dynamic, request-aware routing architecture, the improved system achieves the following capabilities:

  1. Substantial Energy Efficiency Gains: The system actively routes requests away from inefficient static modes to highly optimized paths (e.g., low-precision quantization or batching), resulting in a 51.7% reduction in energy consumption per token compared to a fixed FP16 baseline.

  2. Maximized Throughput and Latency: The system guarantees the fastest possible path for each request type, achieving a 2.10× mean latency speedup on typical synthetic deployment workloads by dynamically selecting the optimal blend of caching, batching, and quantization.

  3. Guaranteed Quality Preservation: Unlike static optimization strategies, this dynamic routing ensures that performance gains are not achieved at the cost of model integrity. By routing benchmark-style requests to quality-preserving modes (like INT8), the system maintains benchmark accuracy within a negligible plus or minus 1.5 percentage-point threshold of the full FP16 baseline.

  4. High Operational Efficiency: The entire decision process—from feature extraction to routing—requires only 0.0096 ms CPU overhead per request, making the control mechanism negligible relative to the massive gains in GPU inference time and power consumption, ensuring it is a practical, deployable solution rather than just a theoretical one.

Abstract

RequestRouter is a lightweight request-boundary controller for reducing the latency and energy cost of single-GPU large language model inference. Rather than serving all requests with one static configuration, the system uses cheap request-level features to select one fixed inference mode per request, including FP16, quantized inference, speculative decoding, prefix caching, continuous batching, and hybrid modes such as GPTQ plus prefix caching and INT8 plus continuous batching. We evaluate RequestRouter using an 8B instruction-tuned language model served through vLLM on NVIDIA A100 GPUs. Across the full-scale A100 evaluation--26,500 fixed-mode evaluations followed by 3,500 online-controller evaluations, for 30,000 measured inference executions in total--the controller achieves a 2.10x mean latency speedup over FP16 and a 0.48x energy ratio on deployment-style workloads. A smaller matched evaluation with repeated measurements provides a controlled statistical check of this result: RequestRouter retains a 1.93x latency speedup (95% CI: 1.88--1.98x) and a 0.523 energy ratio (95% CI: 0.506--0.540), showing that the gains persist under a more tightly controlled protocol. On a separate expanded automatic benchmark evaluation, the routed policy retains 99.6% of FP16 macro accuracy. A 100,000-call CPU microbenchmark measures only 0.00475 ms mean routing overhead (0.00532 ms p99). Thus, simple request-aware routing can recover substantial serving efficiency without retraining or modifying the underlying LLM.

Sources

Related papers