Context-Aware Precision Scheduling for Small Language Models
summary
The gist
The gist: A calibrated predictor for mixed-precision latency and an allocator that uses it to schedule tensor precisions as context grows.
In short
The work introduces a method to manage accuracy and speed for small language models as they process longer text contexts. It develops a calibrated predictor to estimate mixed-precision latency based on execution costs, not just weight size. This predictor feeds into an allocator that strategically chooses the best precision for different model parts (tensors) based on whether the goal is minimizing latency or maximizing accuracy.
Key concepts
- Calibrated Predictor
- This tool estimates how long a mixed-precision calculation will take by looking at actual execution costs across different precisions, rather than just counting the number of bits in the weights. It uses a formula derived from measuring reference latency to predict timing changes as context grows.
- Latency Budget D
- This is the maximum time allowed for processing a given part of the model's computation. The allocator uses this budget to decide which precision setting (e.g., low, medium, or high precision) should be used for each specific tensor unit to stay within the time limit.
- Additive Accuracy Estimate
- This measures how much accuracy (measured by KL divergence between reference and quantized distributions) will change when switching a tensor's precision. This estimate helps the allocator balance the latency constraint against maintaining acceptable model performance.
- Context-Dependent Schedule
- The system doesn't just set one precision; it plans a schedule for future context lengths. It repeats the allocation process as context grows to predict which tensors will need changes and when, allowing it to execute efficiently while keeping the necessary memory (KV cache) intact.
Terminology used across episodes
This episode discusses
- Context-Aware Precision Scheduling for Small Language Models · Paper Radio
- Progressive Mixed-Precision Decoding for Efficient LLM Inference
- HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision
- MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design
- Accuracy is Not All You Need
- KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization
- A KL Lens on Quantization: Fast, Forward-Only Sensitivity for Mixed-Precision SSM-Transformer Models
- KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache
- MorphServe: Efficient and Workload-Aware LLM Serving via Runtime Quantized Layer Swapping and KV Cache Resizing
- HAQ: Hardware-Aware Automated Quantization with Mixed Precision
- LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization
The paper
Context-Aware Precision Scheduling for Small Language Models · Read on arXiv
Mukund Athmaram, Manoj Rao, Liang Ma
Advanced Micro Devices Inc. · Cornell University
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: Today's paper: "Context-Aware Precision Scheduling for Small Language Models".
Jane: The gist: A calibrated predictor for mixed-precision latency and an allocator that uses it to schedule tensor precisions as context grows.
Tom: First, who's behind it and why it matters.
Paper summary: Tom: So this paper is about this Context-Aware Precision Scheduling for Small Language Models. Basically, they’re tackling the problem that when you use small language models on devices, you have to balance how accurate the answer is with how fast it runs as the context gets longer >
Jane: Right, so if you have a big conversation going on and your model has to keep track of all that history, the accuracy can drop or latency can skyrocket depending on what precision you use for different parts of the model >
Lu: The authors claim they’ve calibrated a timing model from measurements taken at different precisions and context lengths, and then they combine that with how sensitive the tensors are to choose configurations under either a latency or an accuracy budget >
Meng: They argue that fewer bits don't always mean faster execution, so their predictor estimates latency from calibrated execution costs instead of just looking at the weight bytes alone >
Lalam: This is about creating a schedule of tensor changes that can be prepared before you even execute everything, which is what they call context-dependent scheduling >
Tom: It seems like they’re trying to figure out exactly which parts of the model should be high precision and which ones can drop down in precision as the context grows >
Jane: And under a latency budget, if you have less time for weight operations because the key value cache is taking up space, then the allocator chooses lower precisions >
Lu: They set up this allocation process by solving for min X u deltaA(u, pu) subject to T0(L) plus X u deltaT(u, pu) being less than or equal to your latency budget D >
Meng: And under an accuracy budget, they approximate the weight and KV cache errors as additive so you can see how much precision can move in either direction >
Lalam: They test this on a Radeon 780M laptop GPU and even extend execution to 16K contexts with Q8, Q6, and Q4 weights > <ref:2610.15640#pg2,extend execution to 16K contexts with Q8, Q6, and Q4>
Conclusion: Tom: So we’ve seen how they build this calibrated predictor for mixed-precision latency in the paper Context-Aware Precision Scheduling for Small Language Models >
Jane: It seems like the authors are showing that they can estimate latency from execution costs rather than just weight bytes, which is a big shift in how you think about model performance on devices >
Lu: They combine this timing change estimation with tensor sensitivities to choose precisions based on your constraints, whether it’s a time budget or an accuracy budget >
Meng: The idea of precomputing which tensors should change and when, while keeping the KV cache active, is what they call executing a context-dependent schedule >
Lalam: And the results show that their cost schedule has sixteen point eight percent lower joint KL and three point three percent lower latency compared to the byte schedule >
Tom: It really shows that this method provides improved latency–fidelity tradeoffs and faster tool-use responses at matching correctness, which is what they found when evaluating tool-use requests with Qwen2 point 5-3B-Instruct > <ref:2610.15640#pg3>
Jane: The paper also points out some limitations, though, like the calibration needing to be done for the specific device and software you are using >
Lu: And they flag that recovery of cache error through weight precision isn't a reliable absolute budget constraint, which means this method doesn't solve every single problem right away >
Meng: So for deployment right now, it’s about improving serving efficiency for lower-cost and private inference, and the hope is that systems can use this predictor to choose precision as a session grows >
More episodes
- 2610.15284-FragFlex: Fragment-Based Flexible Molecular Generation for Exploring Vast Chemical Space
- 2610.15225-Zeroth-Order Fine-Tuning of LLMs in Learnable Low-Rank Subspaces
- 2610.15241-Explicit Bounds on Hoffman Constants for Sparse Polyhedral Regularizers
- 2610.15250-Correlation-Aware Reconstruction for Multivariate Time Series Anomaly Detection
- 2610.15368-ANTE: Anticipate, Then Execute -- Learning Internal World Models for Coding Agents from Verified Beliefs
- 2610.15391-DC-Flow: Target-Conditioned Cyclic Peptide Design via Delayed-Coupling Flow Matching
- 2610.15379-LiveAnon: Improving Privacy and Utility in Streaming Speaker Anonymization with Codec Language Models
- 2610.15097-Anchors Enhance Controllability and Reasoning of Diffusion Language Models
- 2610.15398-GA-SVD: Generation-aware Singular Value Decomposition for LLM Compression
- 2610.15106-RELAGRAM: Recursive Latent Graph Model for the Capacitated Vehicle Routing Problem