speculative decoding

professionalwidespread

Acceleration method: a small, fast model guesses the next parts of the text, and the large model only checks and corrects them if necessary.

In plain terms

Imagine an experienced editor and a fast-typing intern writing together. The intern runs ahead and writes a few words, while the editor glances over them in an instant: what is good stays, and where a mistake is seen, the editor takes over the pen. The text will be just as good as if the editor had written it all along, only it is ready sooner.

Language models typically write answers piece by piece, token by token, and the entire huge model must be run for every token. This is slow. Speculative decoding exploits the fact that most of the text is easily predictable (conjunctions, common phrases, the end of a started word), for which a much smaller model is sufficient.

The method does not sacrifice quality for speed. In the classic version, the response is of the same quality as if the large model worked alone: it is mathematically provable that it selects words with the same probability. Word-for-word, the two texts do not always match because there is inherent randomness in model responses. This distinguishes it from many other acceleration methods that use smaller or compressed models, thereby somewhat degrading the response.

How it works

The system keeps two models ready: a large, precise target model and a small, fast draft model.
The draft model runs ahead and guesses several tokens in sequence, typically a short text segment.
The large model examines all suggestions in a single run, in parallel. This is fast because checking several ready tokens takes roughly the same time for it as writing one.
The large model accepts suggestions as long as they align with its own calculations. At the first rejected suggestion, it inserts its own token and discards subsequent suggestions.
The process restarts from here. The more often the small model guesses right, the greater the speedup; in good cases, the response is produced two to three times faster.

An everyday example

You ask a chatbot to rewrite a long email to be more polite, and the response scrolls noticeably fast across the screen. It is possible that a small model is predicting the predictable parts of the text in the background, while the large model only approves them. You see nothing of this, only that you wait less.

Who is affected

developerIf you run a language model on your own machine or server, this setting speeds up responses without changing the model. → Code

entrepreneur, executiveFaster responses mean better customer experience, but note that the method reduces waiting time, not necessarily computing costs.

everyonePartly due to such background tricks, chatbots and AI agents respond increasingly faster while their quality does not deteriorate.

A common misconception

MythWith speculative decoding, you actually receive the output of a small, weaker model, so the price of speed is lower quality.

In realityThe small model only makes suggestions; the large model decides on every single token. In the classic procedure, the quality of the response is the same as what the large model would give alone.

Limits and risks

The speedup depends on how often the small model guesses right: it can be large for predictable text and small for unusual, creative, or highly specialized text.
Two models must be kept in memory simultaneously, and discarded guesses represent wasted computation, meaning the total work performed may even increase.
On heavily loaded servers processing many requests simultaneously, the advantage may be much smaller than for a single user.
It does not make the model smarter: it can still make mistakes and hallucinate, only it writes faster.

Similar, but not the same

model distillation – In distillation, the small model permanently takes the large model's place and loses some of its knowledge, whereas here the large model continuously monitors and decides.
quantization – Quantization compresses the model itself using less precise number representation, while speculative decoding does not touch the model but reorganizes the text-writing process.
prompt caching – Caching saves the recalculation of previously processed input, while speculative decoding speeds up the writing of new text.

Related concepts

Broader

inference, inference optimization, large language model

Narrower

draft model, self-speculative decoding, predicted output

On the edge

token, model distillation, quantization, latency, small language model

A brief history

The method was described by Google researchers (Yaniv Leviathan, Matan Kalman, and Yossi Matias) in a study published in November 2022. Shortly afterward, in early 2023, DeepMind researchers independently published a very similar procedure under the name speculative sampling. The name refers to the long-known speculative execution in processors, where the computer performs an operation in advance that is likely to be needed.

Where things stand

By the end of 2026, this is a standard engineering solution: several widely used open-source model runners (such as llama.cpp, vLLM, and Hugging Face Transformers) support it natively, and major providers also use it in the background. The user only perceives the faster response.

Status: 2026 November

Also known as

speculative decoding · speculative decoding · speculative sampling · speculative sampling · assisted generation · assisted generation · draft model · draft-model decoding

Sources

This entry was machine-edited and has undergone machine expert review; the “In the news” section updates automatically from Agent Age stories.