DuoRate: Two-Rate Per-Expert Precision Allocation for Mixture-of-Experts Serving
summary
The gist
The gist Serving a Mixture-of-Experts (MoE) model pays for expert precision twice: the resident footprint pays for every expert the router can reach, while each batch-one decode step pays only for
In short
Serving Mixture-of-Experts (MoE) models involves two costs: storing every expert and decoding only the selected ones. DuoRate solves this by storing weights once and reading them per workload using a Lagrangian rate-distortion allocation. This method achieves superior accuracy compared to existing 2-bit baselines, matching 4-bit formats under long decoding conditions.
Key concepts
- Two Rate Functionals
- MoE precision is defined by two rates: the resident rate (how much storage is used for all experts) and the routing-weighted decode rate (how much traffic is generated during inference). The paper shows that optimizing for only one of these rates ignores a significant 'store-read gap' between storage and decoding costs.
- Store-Read Gap
- This gap represents the difference between the cost of storing all experts versus the actual traffic generated by decoding only the selected experts. Published allocators often only constrain the store rate, but this gap is crucial for achieving optimal performance in MoE serving.
- DUORATE Pipeline Stages
- The proposed system has three stages: offline storage where a uniform weight tier and a distortion tensor are baked; online profiling of a workload's prefill without labels; and the final selection stage where it solves an allocation problem using one dot product per expert and a bisection to meet the read budget.
- Routing Frequency Cancellation
- The paper proves that routing frequency does not affect the optimal per-expert decision during decoding. This allows the read-side allocation to be solved independently for each expert, simplifying the process and ensuring efficiency across different workloads.
Terminology used across episodes
This episode discusses
- DuoRate: Two-Rate Per-Expert Precision Allocation for Mixture-of-Experts Serving · Paper Radio
- Progressive Mixed-Precision Decoding for Efficient LLM Inference
- MODE: Modality-Decomposed Expert-Level Mixed-Precision Quantization for MoE Multimodal LLMs · Paper Radio
- MoPEQ: Mixture of Mixed Precision Quantized Experts
- ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving
- SliceMoE: Bit-Sliced Expert Caching under Miss-Rate Constraints for Efficient MoE Inference
- Dynamic Expert Quantization for Scalable Mixture-of-Experts Inference · Paper Radio
- Training Verifiers to Solve Math Word Problems
- GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs
- HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks
- MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design
- Accuracy is Not All You Need
- Measuring Massive Multitask Language Understanding
- Mixture Compressor for Mixture-of-Experts LLMs Gains More
- MC#: Mixture Compressor for Mixture-of-Experts Large Models
- SqueezeLLM: Dense-and-Sparse Quantization
- From Tokens to Layers: Redefining Stall-Free Scheduling for MoE Serving with Layered Prefill
- Mix-Quant: Quantized Prefilling, Precise Decoding for Agentic LLMs
- Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs
- HiFloat4 Format for Language Model Inference
- Pointer Sentinel Mixture Models
The paper
DuoRate: Two-Rate Per-Expert Precision Allocation for Mixture-of-Experts Serving · Read on arXiv
Mengqi Guo, Zhe Wei, Yuan Yuan, Jiunn Bin Lim, Boyi Pan, Michael Bi Mi
Huawei Technologies Ltd.
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: Today's paper: "DuoRate: Two-Rate Per-Expert Precision Allocation for Mixture-of-Experts Serving".
Jane: The gist Serving a Mixture-of-Experts (MoE) model pays for expert precision twice: the resident footprint pays for every expert the router can reach,
Tom: First, who's behind it and why it matters.
Paper summary: Tom: So we're looking at the paper "DuoRate: Two-Rate Per-Expert Precision Allocation for Mixture-of-Experts Serving," which is about managing the storage versus the decoding cost when you use a Mixture-of-Experts model on arXiv #pg6.
Jane: Yeah, they’re tackling those two different rate functionals we talked about earlier, trying to figure out how to handle both the resident footprint and the decode traffic separately.
Lu: The core idea is that published per-expert MoE allocators only constrain one of those rates, but there's a gap between them they call the store–read gap #pg5.
Meng: And they show this difference between storing one expert versus reading many experts at runtime is significant, like two point five four bits per weight becoming seven point five zero bits of decode traffic in one case #pg6.
Tom: So DuoRate proposes a way to fix that by storing it once and then reading it on demand for each specific workload #pg5.
Jane: That sounds like a very practical deployment idea, something you can just ship as one artifact rather than managing separate storage and inference strategies #pg5.
Lu: The key mechanism they use is solving a Lagrangian rate–distortion allocation online, where the routing frequency cancels out in the per-expert decision #pg5.
Tom: I’m still trying to get my head around how that cancellation works without it being overly complicated for production systems #pg5.
Jane: That’s what we want to know—how do you make sure this mathematical trick actually translates into something stable when a model is running?
Meng: From a deployment standpoint, if the allocation can be solved in one dot product per expert and some bisection, that keeps the online cost manageable #pg5.
Lu: The authors also show that routing frequency provably cancels from every per-expert decision within a domain and survives through exactly two channels #pg5.
Conclusion: Tom: We're moving on to the formal math behind this, discussing how they define these two rate functionals in "DuoRate: Two-Rate Per-Expert Precision Allocation for Mixture-of-Experts Serving" #pg5.
Jane: The paper formalizes MoE weight precision as two rate functionals, a resident rate and a routing weighted decode rate #pg5.
Lu: These two rates are defined as Rstore = one over E sigma e R(Mstatic e) for the resident footprint and Rread = sigma e fe R(Me) for the decode traffic #pg5.
Tom: So we have these two rates, and the difference between them is called the store–read gap #pg5.
Jane: The paper proves that routing frequency cancels from every per-expert decision within a domain and survives through exactly two channels #pg5.
Lu: Routing frequency cannot reorder two experts because the per-expert minimizer of the Lagrangian relaxation does not depend on f #pg5.
Meng: That's interesting because usually you worry about routing frequency adding complexity to the cost calculation #pg6.
Tom: So they’re saying we can ignore that complexity when we are making decisions per expert in this way #pg6.
More episodes
- 2610.12566-Sample-Adaptive Zeroth-Order Post-Training
- 2610.12571-An all-transistor physical-learning transformer can be trained to perform in-context learning
- 2610.12694-Critical Mass: Structure-Disjoint Splits Drift With the Libraries They Are Built On
- 2610.12702-Product Monte Carlo: Expectation Under a Product of Densities
- 2610.12697-Toward Maximizing Algebraic Connectivity via Density-Dependent Backbone Construction and Reinforcement Learning Completion
- 2610.12766-Provenance-grouped protocol for evaluating machine learning in perovskite photovoltaics
- 2610.12723-Spatial Context or Memory? Sizing the Inputs of Neural Surrogates for Partially Observed Fields
- 2610.12776-Coordinate-Free Automatic Differentiation
- 2610.12530-CacheScope: Exact-Compute Allocation for Training-Free Diffusion Acceleration
- 2610.12754-NV-Tesseract-AD: Curriculum-Enhanced Conditional Diffusion for Multivariate Time-Series Anomaly Detection