In this post we look at techniques that generate several tokens per forward pass of the model: speculative decoding, where the drafts come from a small model, the context, extra heads, the features or early layers of the model, or a block diffusion model, and multi-token prediction, which builds the drafter into the model during pre-training. As before, all the code is in the repository of this post.

Speculative decoding: more than one token per forward pass

The idea

Decode is memory-bound: a forward pass over one token costs about the same as a forward pass over a handful of tokens, because the time goes into reading the weights. Speculative decoding exploits this. A cheap method drafts \(K\) tokens, and the large target model checks all of them in a single forward pass. Every draft token the target agrees with is a token generated almost for free. When the target disagrees, we keep the tokens before the disagreement, take the target’s own token at the disagreement, and draft again. With the right acceptance rule, the output is exactly the same as with the target model alone: speculative decoding is lossless.

If each draft token is accepted independently with probability \(\alpha\), the expected number of tokens produced per forward pass of the target model, including the target’s own token after the last accepted draft, is:

\[\frac{1 - \alpha^{K+1}}{1 - \alpha}\]

Drafting is not free. If drafting one token costs a fraction \(c\) of a forward pass of the target, and a verification costs about as much as a regular decoding step (decode is memory-bound), the expected speedup over decoding with the target alone is (Leviathan et al.):

\[\frac{1 - \alpha^{K+1}}{(1 - \alpha)(cK + 1)}\]

The speedup therefore depends on how often the drafts are right and on how cheap they are. Speculative decoding helps most at small batch sizes, where decode is most memory-bound; at large batch sizes, the GPU is already busy, and the extra verification work competes with useful work. The methods below differ in where the drafts come from.

Draft-model speculative decoding

The original method (Leviathan et al., Chen et al.) drafts with a small language model that uses the same tokenizer as the target, ideally a distilled version of the target, such as the student we train in the distillation section of a previous post. Each iteration:

  1. the draft model generates \(K\) tokens autoregressively;
  2. the target model runs one forward pass over the context and the \(K\) draft tokens, which gives its distribution at each of the \(K\) draft positions, plus one more after the last draft;
  3. the draft tokens are checked from left to right.

The acceptance rule depends on how we decode:

  • With greedy decoding, draft tokens are accepted while they are equal to the target’s most likely token. At the first mismatch, the draft token is replaced by the target’s most likely token, and the remaining drafts are discarded. If all \(K\) drafts are accepted, the target’s prediction after the last draft is appended as a bonus token. The output is identical to greedy decoding with the target alone.
  • With sampling (speculative sampling), the draft model samples each token \(x\) from its distribution \(p\), and the target accepts it with probability \(\min\left(1, q(x) / p(x)\right)\), where \(q\) is the target’s distribution. On rejection, a replacement token is sampled from the normalized residual distribution \(\max(0, q - p)\), and the remaining drafts are discarded. This modified rejection sampling produces tokens distributed exactly as \(q\), and if all \(K\) drafts are accepted, a bonus token is sampled from the target’s distribution after the last draft. The rule holds for any \(p\), as long as it is the distribution the drafts were actually sampled from. Greedy drafts are the special case of a one-hot \(p\): the target accepts the draft \(x\) with probability \(q(x)\), and on rejection samples from \(q\) without \(x\), which still yields tokens distributed as \(q\). What breaks the guarantee is drafting greedily while using the draft model’s softmax probabilities as \(p\).

In a batch, each sequence accepts a different number of tokens, so the sequences progress at different speeds, and engines track their lengths separately, as in continuous batching.

To keep it simple, our implementation generates one sequence at a time, and the whole sequence must fit in the context window. Both models use a KV cache: after each verification, the cache of the target is truncated to the accepted tokens, and the draft model reuses the part of its own cache that is still valid.

The following code implements greedy speculative decoding in GPTlite, with the distilled model as the draft. The full code is in speculative_decoding.py, which checks that the output is the same as greedy decoding with the target alone:

Show code 
def generate_speculative(model, prompt, n_tokens, propose, n_drafts):
  """ Greedy speculative decoding of a single sequence (prompt of shape [1, S]). At every iteration,
      propose(tokens, k) returns up to k draft tokens, and the target model checks all of them in a single
      forward pass. The cache of the target holds all the accepted tokens except the last one, which is
      processed together with the drafts. The whole sequence must fit in the context window. """
  assert prompt.size(1) + n_tokens <= model.seqlen, "the sequence must fit in the context window"
  tokens, kv_cache = prompt, None
  if prompt.size(1) > 1:
    _, kv_cache = model(prompt[:, :-1])
  stats = {"target forward passes": 0, "proposed drafts": 0, "accepted drafts": 0}
  while tokens.size(1) < prompt.size(1) + n_tokens:
    drafts = propose(tokens, min(n_drafts, model.seqlen - tokens.size(1)))  # [1, k]
    # target predictions after the last accepted token and after each draft: [1, k+1]
    logits, kv_cache = model(torch.cat([tokens[:, -1:], drafts], dim=1), kv_cache=kv_cache)
    predictions = torch.argmax(logits, dim=-1)
    n_accepted = 0  # accept the drafts while they match the predictions of the target
    while n_accepted < drafts.size(1) and drafts[0, n_accepted] == predictions[0, n_accepted]:
      n_accepted += 1
    # keep the accepted drafts, plus the target's own token at the first mismatch (or the bonus token, after
    # the last draft if all were accepted), and discard the cache entries of the rejected drafts
    tokens = torch.cat([tokens, drafts[:, :n_accepted], predictions[:, n_accepted:n_accepted+1]], dim=1)
    kv_cache = truncate_cache(kv_cache, tokens.size(1) - 1)
    stats["target forward passes"] += 1
    stats["proposed drafts"] += drafts.size(1)
    stats["accepted drafts"] += n_accepted
  return tokens[:, :prompt.size(1) + n_tokens], stats

class DraftModel:
  """ Proposes draft tokens with a small model and its own KV cache. On every call, it reuses the cache
      of the longest prefix of the accepted tokens it has already processed """

  def __init__(self, model):
    self.model = model
    self.kv_cache, self.processed = None, None  # cache and the tokens it holds

  def __call__(self, tokens, n_drafts):
    if n_drafts == 0:
      return tokens[:, :0]
    n_cached = 0
    if self.processed is not None:  # number of leading tokens processed before and still valid
      n = min(self.processed.size(1), tokens.size(1) - 1)
      mismatches = (self.processed[0, :n] != tokens[0, :n]).nonzero()
      n_cached = mismatches[0].item() if len(mismatches) > 0 else n
    kv_cache = truncate_cache(self.kv_cache, n_cached) if n_cached > 0 else None
    new_tokens, drafts = tokens[:, n_cached:], []
    for _ in range(n_drafts):  # greedy generation with the draft model
      logits, kv_cache = self.model(new_tokens, kv_cache=kv_cache)
      new_tokens = torch.argmax(logits[:, -1:], dim=-1)  # [1, 1]
      drafts.append(new_tokens)
    drafts = torch.cat(drafts, dim=1)
    self.kv_cache, self.processed = kv_cache, torch.cat([tokens, drafts[:, :-1]], dim=1)  # the last draft was not processed
    return drafts

Prompt lookup decoding

Prompt lookup decoding needs no draft model at all. It takes the last few generated tokens (an n-gram), searches for the same n-gram earlier in the prompt or in the generated text, and proposes the tokens that followed it as drafts; the verification is the same as above. It costs almost nothing, and gives large speedups when the output copies from the input, as in summarization, code editing, question answering over documents, or rewriting a previous answer. Related methods draft from other sources: lookahead decoding collects n-grams from Jacobi iterations of the model itself, and REST retrieves continuations from a datastore of text.

The following code implements prompt lookup decoding, reusing the verification loop of speculative decoding with a different source of drafts. The full code is in prompt_lookup.py:

Show code 
def prompt_lookup(tokens, n_drafts, max_ngram=3):
  """ Drafts the tokens that followed the most recent earlier occurrence of the last n-gram of the sequence,
      trying the longest n-grams first. Returns no drafts if the n-gram never occurred before """
  sequence = tokens[0]
  for n in range(max_ngram, 0, -1):
    if sequence.size(0) <= n:
      continue
    ngram = sequence[-n:]
    windows = sequence[:-1].unfold(0, n, 1)  # all the earlier n-grams of the sequence
    matches = (windows == ngram).all(dim=1).nonzero()
    if len(matches) > 0:
      start = matches[-1].item() + n  # first token after the most recent match
      return sequence[start:start + n_drafts].unsqueeze(0)
  return tokens[:, :0]

# usage: the same verification loop as speculative decoding, with a different source of drafts
tokens, stats = generate_speculative(model, prompt, n_tokens, prompt_lookup, n_drafts)

Medusa

Medusa adds a few extra decoding heads on top of the last hidden state of the target model: the original output head predicts the next token, and head \(i\) predicts the token \(i+1\) positions ahead. The top candidates of each head are combined into a tree of possible continuations, and all of them are verified in a single forward pass with tree attention, a mask that lets each candidate attend only to its own ancestors in the tree. The longest accepted path is kept. Training only the heads, with the model frozen (Medusa-1), gives speedups above 2.2 times; training the heads together with the model (Medusa-2) reaches 2.3 to 3.6 times. Medusa needs no separate draft model, but each head guesses its token independently, without seeing the tokens guessed by the other heads, so the accuracy drops quickly for later positions. For sampling, Medusa also proposes a faster typical acceptance rule, which is not exactly lossless.

EAGLE

EAGLE drafts at the level of features rather than tokens. Its draft model is a single lightweight transformer layer that takes the top-layer hidden states (features) of the target, together with the embeddings of the tokens sampled so far, predicts the next feature, and turns it into a token with the target’s own output layer. Feature sequences are more regular than token sequences, so these drafts are much more accurate than Medusa’s independent heads. EAGLE-2 shapes the draft tree dynamically, expanding the branches where the draft model is confident. EAGLE-3 (NeurIPS 2025) noticed that EAGLE barely improved with more training data, and traced this to its feature prediction objective. It drops feature prediction and predicts tokens directly, fuses low-, mid- and high-level features of the target instead of the top layer only, and uses training-time test: during training, the draft model is fed its own predictions, to simulate multi-step drafting. EAGLE-3 reaches speedups of up to 6.5 times, about 1.4 times more than EAGLE-2, and 1.38 times higher throughput at a batch size of 64 in SGLang. EAGLE-style drafting is supported by vLLM, SGLang and TensorRT-LLM.

DFlash

DFlash (Chen, Liang and Liu, UC San Diego, ICML 2026) replaces the autoregressive drafter with a small block diffusion model. Autoregressive drafters such as EAGLE-3 need one forward pass per drafted token, so they are kept very shallow (a single transformer layer for EAGLE-3), which, according to the authors, limits their acceptance length and keeps practical speedups around 2-3 times. DFlash instead takes the last verified token, appends 15 mask tokens, and fills the whole block in a single forward pass, so its drafting cost barely grows with the block size (in the speedup formula above, \(cK\) becomes about \(c\)) and it can afford a deeper drafter: a 5-layer DFlash fills a 16-token block faster than a 1-layer EAGLE-3 drafts 8 tokens, and gets more of them accepted.

A drafter this small predicts poorly on its own (without target features, a 5-layer diffusion drafter only reaches about 3 times), so DFlash conditions it on the target: the hidden states of 5 target layers, spread from shallow to deep, are concatenated, projected, and injected into the keys and values of every draft layer, instead of only into the drafter’s input as in EAGLE-3, so the acceptance length keeps improving as draft layers are added. The drafter reuses the target’s frozen embedding and output head, and only its transformer layers are trained, on responses generated by the target: each training block starts at a randomly sampled anchor token, as at inference, all blocks of a sequence are trained in one pass with a block-sparse attention mask, and the loss favors early positions, since one early mistake invalidates the rest of the block. Verification is unchanged, so the output stays lossless. Each round verifies a single drafted block, not a tree: DDTree builds a draft tree from the per-position predictions of the DFlash drafter, and verifies it with tree attention.

On Qwen3-8B with greedy decoding, the paper reports about 6.5 tokens committed per verification step on average and up to 6.1 times faster decoding than the baseline, about 2.5 times the speedup of EAGLE-3, with the largest gains on math and code and the smallest on open-ended chat (about 2.8 times). In SGLang on a B200, the speedup drops from about 4-5 times for a single request to about 2.5-3 times at 32 concurrent requests, as verifying 16-token blocks becomes compute-bound. Drafters trained on 4K-token contexts also lose acceptance on longer inputs, which a short long-context fine-tune recovers. Unlike multi-token prediction (below), which only helps models pre-trained with MTP modules, a DFlash drafter is trained after the fact against a frozen target, so it can be added to almost any model. However, at the time of writing its training recipe is not public, so in practice one uses the drafters released by z-lab, for Qwen3.5/3.6, gpt-oss, Gemma 4 and Kimi K2.5, among others. DFlash runs in vLLM and SGLang, with an MLX version for Apple Silicon.

Self-speculative decoding: LayerSkip

LayerSkip uses the early layers of the target model as the draft. The model is trained with layer dropout, which skips the later layers more often, and with an early exit loss, which trains every layer to make good predictions through the shared output head. At inference, the draft exits after the first \(E\) layers, and the verification runs only the remaining layers, reusing the computation and the cache of the first \(E\). There is no second model to store, and the reported speedups reach up to about 2 times on summarization and coding tasks.

Multi-token prediction

Multi-token prediction (MTP) trains the model itself, during pre-training, to predict several future tokens at every position, instead of only the next one. This has two benefits. During training, every position gets a denser learning signal, which can improve the representations learned by the model. At inference, the extra predictions are drafts: the model drafts for itself, without a separate draft model, and the verification is the same as in speculative decoding.

Parallel heads

Gloeckle et al. (Meta, 2024) add \(n\) output heads on top of a shared trunk. The trunk \(f_s\) computes a representation \(z_t = f_s(x_{1:t})\) of the context, and head \(i\), a transformer layer \(f_{h_i}\) followed by the unembedding \(f_u\) shared by all heads, predicts the token \(i\) positions ahead:

\[P_\theta(x_{t+i} \mid x_{1:t}) = \text{softmax}\left( f_u \left( f_{h_i} ( z_t ) \right) \right)\]

The training loss sums the cross-entropy losses of the \(n\) heads:

\[L_n = - \sum_t \sum_{i=1}^{n} \log P_\theta(x_{t+i} \mid x_{1:t})\]

As the vocabulary is large, holding the logits of all heads at once would dominate the memory of training. Instead, after the forward pass of the trunk, the forward and backward passes of the heads run one after the other, accumulating their gradients at the output of the trunk, so that only the logits of one head are in memory at a time. The benefits grow with the model size, and small models can even get worse: the 13B model of the paper solves 12% more problems on HumanEval and 17% more on MBPP than a comparable next-token model. At inference, the extra heads are drafts for self-speculative decoding, making a model trained to predict 4 tokens up to 3 times faster. However, as in Medusa, each head predicts its token without seeing the tokens predicted by the other heads.

Sequential modules

DeepSeek-V3 predicts the extra tokens sequentially instead, keeping the complete causal chain at each prediction depth. It adds \(D\) MTP modules, where module \(k\) has its own transformer block \(\text{TRM}_k\) and projection matrix \(M_k\), and shares the embedding layer and the output head with the main model. For the token at position \(i\), module \(k\) combines the representation \(h_i^{k-1}\) of the previous depth (the output of the main model, for \(k=1\)) with the embedding of the token \(t_{i+k}\):

\[h_i^{\prime k} = M_k \left[ \text{RMSNorm}(h_i^{k-1}) ; \text{RMSNorm}(\text{Emb}(t_{i+k})) \right]\]

runs it through its transformer block, \(h^{k} = \text{TRM}_k(h^{\prime k})\), and predicts the token \(t_{i+k+1}\) with the shared output head, \(P^{k}_{i+k+1} = \text{OutHead}(h^{k}_i)\). Each depth has its own cross-entropy loss \(L^k_\text{MTP}\), and their average is added to the main next-token loss with a weight \(\lambda\):

\[L_\text{MTP} = \frac{\lambda}{D} \sum_{k=1}^{D} L^k_\text{MTP}\]

DeepSeek-V3 uses a single module (\(D=1\)), with \(\lambda = 0.3\) for the first 10T training tokens and \(0.1\) for the remaining 4.8T, and its published MTP weights add 14B parameters to the 671B of the main model.

  • Sequential / cascaded modules — NOT parallel towers. This is the defining detail. The main model predicts \(t_{i+1}\) from the shared trunk. Then MTP module 1 (a single lightweight Transformer layer) takes the trunk’s hidden states plus the embedding of the just-predicted token to predict \(t_{i+2}\); MTP module 2 chains off module 1 to predict \(t_{i+3}\); and so on. Each module depends on the previous one, preserving the full causal chain. (Gloeckle et al.’s original MTP uses parallel independent heads off one trunk — V3 deliberately chose the sequential version to model inter-token dependencies.)
  • Shared embedding & output head. Every MTP module reuses the main model’s embedding and unembedding/output head — only the small per-module Transformer layer is extra. Open-source V3 ships 1 MTP module (depth = 1).
  • Auxiliary training loss. Each module’s future-token prediction carries its own cross-entropy loss, added (weighted) to the main loss. Penalizing bad “look-ahead” guesses forces the trunk to build representations useful for longer-range planning — helping coherence in reasoning, code, and math.
  • Inference: default single-token, optional speculative decoding. By default only the main head runs (max accuracy). But the MTP modules can act as a built-in draft model: they propose the next few tokens, and the main model verifies them in parallel in one forward pass. V3’s MTP-1 has an ~80–90% acceptance rate, giving ~1.8× higher generation throughput.

The authors report better results on most benchmarks, and suggest that MTP may make the model pre-plan its representations for better predictions of future tokens. At inference, the module can be discarded, and the model runs as a regular next-token model, or used as a draft: the second predicted token is accepted 85-90% of the time, giving 1.8 times more tokens per second. Note that an MTP module is very close to an EAGLE draft layer: both combine the features of the target with the embedding of the next token, run one transformer layer, and predict with the target’s output head. The difference is that the MTP module is trained together with the model, during pre-training.

Drafting with MTP modules

With a single module, the model drafts one extra token per forward pass. To draft more tokens, inference engines run the same module again on its own output hidden state and draft token, as with EAGLE, even though it was only trained to look one token ahead, so the acceptance drops for the later drafts. SGLang serves DeepSeek-V3 and R1 this way, with their MTP module (called NextN) as the drafter: in a large-scale deployment, LMSYS reported an average acceptance length of 2.18 tokens with a 3-token MTP window, and of 2.44 tokens with a 4-token window. Other models train more modules for inference: MiMo-7B (Xiaomi) uses a single MTP layer during pre-training, as more layers did not help, and then copies it to fine-tune two more MTP layers, with the main model and the first MTP layer frozen. On AIME24, the drafts of the first MTP layer are accepted about 90% of the time, and those of the third layer more than 75%. Native MTP modules are now common in recent models, such as Qwen3-Next and Qwen3.5: vLLM uses them as drafters, and its speculators library can fine-tune them for multi-step drafting.

Medusa, EAGLE and DFlash add a drafter to an existing model, and LayerSkip trains the model so that its own early layers can draft. MTP builds the drafter in during pre-training: its drafts are cheap and accurate, and it may even improve the model, but only models pre-trained with it can use it.

Summary of speculative decoding methods

Method Source of the drafts Extra training Extra memory Reported speedup
Draft model a separate small model none, or distilling a draft a second model about 2-3x
Prompt lookup n-grams of the context none none large on copy-heavy tasks
Medusa extra heads the heads (or joint fine-tuning) small 2.2-3.6x
EAGLE-3 a feature-level draft layer the draft layer small up to 6.5x
DFlash a block-diffusion drafter conditioned on the target’s hidden states the drafter (target frozen) small (a few layers) over 6x, about 2.5x EAGLE-3’s speedup
LayerSkip early layers of the model layer dropout and early exit loss, in pre-training or fine-tuning none up to about 2x
Multi-token prediction the model’s own MTP heads (parallel) or modules (sequential) pre-training small (about 2% of DeepSeek-V3) 1.8x (DeepSeek-V3) to 3x

The reported speedups come from the respective papers, with different models, tasks and hardware, and they shrink as the batch size grows. All these methods are lossless, except Medusa with typical acceptance.