MineDraft: A Framework for Batch Parallel Speculative Decoding

arXiv:2603.18016 · cs.CL, cs.AI, cs.DC, cs.LG · Submitted 2026-02-24 · Read on arXiv

cs.CL, cs.AI, cs.DC, cs.LG

Submitted: 2026-02-24

Updated: 2026-09-01

Comments: Accepted at ICML 2026

Code: https://github.com/huggingface/transformers

License: http://creativecommons.org/licenses/by/4.0/

The gist: Speculative decoding (SD) accelerates large language model inference by using a smaller draft model to propose draft tokens that are subsequently verified by a larger target model.

Terminology

Abstract

Speculative decoding (SD) accelerates large language model inference by using a smaller draft model to propose draft tokens that are subsequently verified by a larger target model. However, the performance of standard SD is often limited by the strictly sequential execution of these drafting and verification stages. To address this, this paper proposes MineDraft, a batch parallel speculative decoding (PSD) framework designed to effectively hide drafting latency by overlapping it with verification. Our theoretical analysis shows that PSD is substantially more efficient than standard SD. MineDraft realizes the PSD through a novel batch-parallel design that maintains two batches of requests, overlapping drafting for one batch with verification for the other. Our experimental results show significant improvements of in both throughput (up to 75%) and end-to-end latency (up to 39%) over standard SD. Furthermore, we have implemented MineDraft as a plugin for vLLM, demonstrating its practicality for production-ready inference systems.

Sources

Related papers