Speculative Decoding: Medusa, MTP, EAGLE, DFlash, DDTree, and DSpark

Speculative decoding changes the unit of progress in LLM inference: a target-model forward pass can commit several tokens instead of one. The difficult part is making those extra tokens cheaper than ordinary decoding after accounting for the drafter, verification, and serving load.

This article compares Medusa, multi-token prediction (MTP), EAGLE-1/2/3, DFlash, DDTree, and DSpark through that systems lens, then maps them to vLLM and SGLang. Sources checked on September 6, 2026. Commands are documentation- and source-checked starting points; no GPU performance measurements were performed for this article. Framework support refers to the inspected upstream snapshots, not every released wheel or hardware backend.

1. What Speculative Decoding Actually Guarantees

#

A small drafter proposes a continuation. The target evaluates its positions together, using causal attention, and accepts a prefix. After a rejection, later draft tokens are discarded: their target probabilities were computed under the rejected prefix.

For example, a draft proposes A B C D. If A B survive and C is rejected, this round can commit A B R, where R is the correction. D cannot be kept merely because it looked plausible.

For standard speculative sampling, let \(p_i\) be the target distribution and \(q_i\) the actual proposal distribution at the same prefix. A sampled draft token \(y_i\) is accepted with probability

\[ \alpha_i(y_i)=\min\left(1,\frac{p_i(y_i)}{q_i(y_i)}\right). \]

At the first rejection, sample from the normalized positive residual \([p_i-q_i]_+\). If every draft token survives, sample an additional token from the target. This construction preserves the target sampling distribution. It does not require the drafter to be equally capable. See Leviathan et al..

Three distinctions matter:

• Greedy equivalence: accepted tokens follow the target’s argmax decisions, subject to numerical implementation differences.
• Distributional equivalence: stochastic outputs follow the same target distribution; they need not match token-for-token under the same random seed.
• Similar quality: an approximate acceptance rule may maintain benchmark quality without preserving the distribution.

Tree verification needs its own correct sampling procedure; the single-chain formula above is not a complete tree algorithm. Temperature, truncation, penalties, and grammar constraints must also be incorporated consistently. A confidence score can determine how much to verify, but cannot replace target verification.

2. The Cost Model Behind the Comparison

#

Let \(A\) be the accepted draft-prefix length, excluding the target correction or bonus token. Ignoring early stopping at EOS and output limits, a round advances \(A+1\) tokens. A useful steady-state approximation is

\[ T_{\mathrm{token}}\approx \frac{\mathbb{E}[T_{\mathrm{draft}}+T_{\mathrm{verify}}+T_{\mathrm{overhead}}]} {\mathbb{E}[A]+1}. \]

This extends the latency accounting in DFlash by explicitly including scheduling and cache-management overhead. At low concurrency, unused compute can make parallel verification attractive. At high concurrency, speculative positions compete with other requests for compute and memory. One verification pass is not necessarily as cheap as one ordinary decode pass.

An illustrative calculation, not a benchmark: if ordinary decoding costs 10 ms/token and a speculative round costs 3 ms drafting + 13 ms verification + 2 ms overhead, accepting three draft tokens gives \(18/4=4.5\) ms/token. Accepting only one gives \(18/2=9\) ms/token. The same implementation can therefore be excellent or barely worthwhile depending on acceptance.

The methods below improve different parts of this cycle: generating proposals, selecting candidate paths, or allocating verification work.

3. Medusa and MTP: Predicting Beyond the Next Token

#

Medusa: Parallel Heads

#

Medusa attaches additional prediction heads to the target’s final hidden state. Different heads predict different future offsets in parallel; their candidates can be assembled into a tree and verified with tree attention. The heads are cheap, but later positions do not condition on the actual tokens selected by earlier heads.

Medusa-1 trains the heads with the backbone frozen. Medusa-2 trains them jointly with the backbone, so the resulting target is no longer exactly the original checkpoint. The paper also offers typical acceptance, which deliberately relaxes distribution matching. “Medusa is lossless” therefore needs an explicit target checkpoint and acceptance policy. See the Medusa paper.

MTP: A Family of Training Objectives and Architectures

#

MTP is broader than one inference algorithm. Gloeckle et al. train a shared trunk with future-token prediction heads. Those auxiliary predictions can later serve as proposals, but merely training with an MTP objective does not automatically accelerate serving.

DeepSeek-V3 uses sequential MTP modules: each combines an earlier representation with the embedding of a shifted token, then applies a Transformer block. Embeddings and the output head are shared. This preserves a causal dependency across prediction depths rather than independently guessing all future positions. See DeepSeek-V3, Section 2.2.

In deployment, distinguish the number of trained MTP modules from the number of speculative tokens requested. Some implementations reuse a module over multiple draft steps. Increasing the latter can increase serial work; it does not create additional trained heads. Native MTP also requires compatible weights and a model-specific loader, not just a CLI switch.

4. EAGLE-1, EAGLE-2, and EAGLE-3

#

EAGLE-1: Predict Features, Condition on Tokens

#

EAGLE moves drafting into the target’s hidden-feature space. It conditions feature prediction on an already sampled, one-step-ahead token, resolving ambiguity about which continuation the feature should represent. A lightweight autoregressive draft network then produces candidates using the target’s output head. Its original verification tree is fixed. See EAGLE.

The important distinction from Medusa is the evolving draft state: future proposals depend on earlier draft choices. The cost is sequential draft-network execution as the proposed path grows.

EAGLE-2: Spend the Tree Budget Dynamically

#

EAGLE-2 retains the draft-model approach and changes candidate construction. It expands promising paths using cumulative draft confidence, then reranks candidates to choose the verification tree. This adapts depth and breadth to the current context instead of repeatedly applying one fixed tree. See EAGLE-2.

It improves the use of a node budget; it does not make tree nodes free. At high load, a wider tree can increase acceptance while worsening total serving throughput.

EAGLE-3: Improve the Learning Problem

#

EAGLE-3 removes the explicit target-feature regression constraint and trains for token prediction. It fuses low-, middle-, and high-layer target features, and uses training-time test to expose training to the drafter’s own multi-step states. This addresses the mismatch between clean training features and recursively generated inference states. It can still use EAGLE-2-style trees. See EAGLE-3.

EAGLE-3 remains autoregressive in drafting. Its advance is better use of features and training data, not one-pass generation of an entire block. A framework loading an EAGLE-family checkpoint also does not prove that it reproduces every tree policy from the papers.

5. DFlash, DDTree, and DSpark

#

DFlash: One Parallel Draft Forward

#

DFlash uses a lightweight block-diffusion drafter. An anchor token and masked future positions are processed together; fused target-layer features condition the draft layers through KV injection. The original draft block is generated in one forward pass, rather than a long iterative denoising loop. The target remains autoregressive and verifies the proposals. See the paper and author implementation.

This trades repeated small draft passes for a wider parallel computation. However, jointly computed hidden states are not the same thing as conditioning on already sampled tokens within the block. Incompatible choices at different positions can shorten the accepted suffix. Larger blocks still consume attention, logits, and cache resources.

DDTree: Build a Tree from Parallel Marginals

#

DDTree reuses a block drafter such as DFlash and constructs multiple candidate paths from its per-position distributions. A prefix \(u\) is scored with the factorized surrogate

\[ Q(u)=\prod_{i=1}^{|u|}q_i(u_i). \]

A best-first heap selects prefixes under a node budget. The resulting tree uses depth-based positions and ancestor-only attention. Verification walks the target’s selected tokens; when no child matches, that target token becomes the next bonus token. Only the accepted path’s KV entries survive. The claimed tree optimality is for the factorized surrogate, not the unknown target distribution. See DDTree.

DDTree changes the candidate set without requiring a newly trained drafter. Its practical question is whether longer acceptance pays for construction, extra verification positions, and KV compaction. The official implementation is a research benchmark, not evidence of a built-in vLLM or SGLang method.

DSpark: Dependencies Plus a Verification Budget

#

DSpark combines a parallel backbone with a lightweight sequential output component, such as a low-rank Markov head or an RNN head. Expensive feature computation stays parallel; token selection gains intra-block dependency. A separate confidence head estimates conditional acceptance, and calibrated prefix-survival estimates guide a hardware-aware scheduler. See DSpark.

For conditional confidence estimates \(c_i\), the estimated probability of accepting through position \(j\) is \(s_j=\prod_{i=1}^{j}c_i\). Verifying a prefix of length \(k\) therefore has estimated progress \(1+\sum_{j=1}^{k}s_j\). This makes the scheduler’s question concrete: is another verification position worth its marginal cost?

In serving, shortening the logical window only helps if the engine actually executes less work. SGLang’s compact, variable-length verification and CUDA-graph handling address that requirement; vLLM also provides confidence-driven adaptive verification. See the SGLang integration and vLLM adaptive verification guide.

6. A Mechanism-Level Comparison

#

This table synthesizes the papers above. “Main constraint” is an engineering interpretation, not a measured ranking.

There is no single succession in which every new method replaces the previous one. EAGLE-2 and DDTree chiefly decide which alternatives to verify. DSpark additionally decides how much verification each request deserves under load. Native MTP remains a useful baseline when compatible weights already ship with the target.

What Published Speedups Do and Do Not Say

#

These are author-reported results from the linked papers, not a common benchmark suite. Comparing the numbers across rows would mix models, hardware, workloads, tuning budgets, and baseline implementations.

7. What vLLM and SGLang Support

#

The source snapshots used here are vLLM f2e2936f91a7 and SGLang febb36051987. These are inspection references, not GPU-tested release recommendations.

Evidence: vLLM configuration, SGLang algorithm registry and dispatch, and the official guides linked below. Absence from this matrix is not a claim about third-party plugins or forks.

Two details prevent misleading comparisons. First, the inspected vLLM Medusa proposer stacks each head’s argmax into a sequence; that is not the original paper’s multi-branch tree construction. Second, SGLang’s general guide does not yet list DSpark, while its worker and official integration article do. Documentation, source, and installed packages can move at different speeds.

8. Starting with vLLM

#

Install a build supporting the selected model and method, then record vllm --version, the container digest or Git SHA, and model revisions. These examples assume sufficient GPU memory; adjust tensor parallelism and context limits for the machine. Gated Llama checkpoints require model access.

EAGLE-3

#

Use a draft trained for the exact target. This pair is identified by the draft model card; the configuration interface is documented in the EAGLE guide.

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --host 127.0.0.1 --port 8000 \
  --max-model-len 8192 \
  --speculative-config '{"method":"eagle3","model":"RedHatAI/Llama-3.1-8B-Instruct-speculator.eagle3","num_speculative_tokens":3}'

For a compatible native-MTP target, the corresponding configuration is {"method":"mtp","num_speculative_tokens":1}. This is a different model path, not something to add to the Llama example. Follow the target’s recipe and the MTP guide.

DFlash and DSpark

#

The method/model interface can be expressed as a template:

# Set TARGET_MODEL and DRAFT_MODEL to a verified matching pair.
vllm serve "$TARGET_MODEL" \
  --host 127.0.0.1 --port 8000 \
  --speculative-config "{\"method\":\"dflash\",\"model\":\"$DRAFT_MODEL\",\"num_speculative_tokens\":$DRAFT_TOKENS}"

DRAFT_TOKENS must match the checkpoint’s block/anchor convention and runtime constraints. For DSpark, use method: "dspark" with a DSpark checkpoint. Public matched Qwen3-8B examples are deepseek-ai/dflash_qwen3_8b_block7 and deepseek-ai/dspark_qwen3_8b_block7, listed in DeepSpec. Its released Qwen drafts were trained in non-thinking mode, which matters when selecting a workload. Check architecture metadata: the DFlash checkpoint can use a DSpark-compatible model class with its Markov/confidence components disabled, so names alone are insufficient.

A DSpark speculative configuration with adaptive verification has this shape:

{
  "method": "dspark",
  "model": "deepseek-ai/dspark_qwen3_8b_block7",
  "num_speculative_tokens": 7,
  "draft_sample_method": "probabilistic",
  "enable_adaptive_verification": true
}

Adaptive verification is off by default. The checked vLLM guide requires a confidence head, compatible attention, and full CUDA graphs; it excludes eager execution, LoRA, and pipeline parallelism for this path. Enabling dspark alone does not enable adaptive verification. Use the adaptive verification guide for the complete hardware-specific launch recipe.

9. Starting with SGLang

#

EAGLE-3 and Native MTP

#

This EAGLE-3 pair follows the SGLang guide:

python3 -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --speculative-algorithm EAGLE3 \
  --speculative-draft-model-path jamesliu1/sglang-EAGLE3-Llama-3.1-Instruct-8B \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 4 \
  --speculative-num-draft-tokens 16 \
  --mem-fraction-static 0.7 \
  --host 127.0.0.1 --port 30000

Here num-steps controls draft depth, eagle-topk controls candidate breadth, and num-draft-tokens controls verification capacity. Sixteen verification positions do not promise sixteen accepted output tokens. The draft topk is also separate from request sampling top_k.

For a supported native-MTP target, such as the guide’s XiaomiMiMo/MiMo-7B-RL, the short-path settings are EAGLE, one draft step, top-k one, and two verification slots. The EAGLE label selects serving machinery; it does not mean the MTP checkpoint was trained using the EAGLE paper.

DFlash

#

python3 -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path z-lab/LLaMA3.1-8B-Instruct-DFlash-UltraChat \
  --host 127.0.0.1 --port 30000

Let the supported runtime infer the initial block size from the checkpoint. Do not transfer EAGLE depth/breadth settings to a linear DFlash block. Check the installed version’s scheduling and parallelism restrictions; the overview document and newer source do not describe exactly the same support surface.

DSpark

#

SGLang uses --speculative-algorithm DSPARK. The official reproduction recipe uses deepseek-ai/DeepSeek-V4-Flash-DSpark on H200 with four-way data-parallel attention, and provides an image and pinned implementation. It is a multi-GPU recipe, not a drop-in single-card example.

For that documented recipe, SGLANG_RAGGED_VERIFY_MODE=static verifies the full window; compact executes selected per-request windows; cap-accept retains a full verification pass for diagnostic comparison while limiting committed progress. The SPS cost-table option is --speculative-dspark-sps-table-path. Profile a table for the deployment rather than copying another GPU’s costs. The distinction between running a DSpark drafter and exercising calibrated, cost-aware trimming should be visible in benchmark labels.

10. Benchmark the Serving System

#

The following is a proposed evaluation protocol, not a report of measurements:

1. Establish a non-speculative baseline in each engine with identical target weights, precision, prompts, context lengths, and sampling settings.
2. Separate chat, coding, and reasoning traffic. Include long-context requests and the expected production mix; keep thinking mode and chat templates fixed.
3. Sweep concurrency, for example 1, 8, 32, and 128. Also sweep arrival rate to expose queueing under an open-loop workload.
4. Sweep a small set of valid draft lengths or node budgets. Compare DSpark with and without adaptive verification. Record graph, attention-backend, and scheduler settings.
5. Report TTFT, inter-token latency/TPOT, total output tokens/s, p50/p95/p99 latency, accepted draft length, verified positions per round, and peak memory. Define whether acceptance metrics include the bonus token.
6. Compare quality separately: greedy token agreement where numerically appropriate, task evaluation, and distribution-sensitive checks for stochastic sampling. Same-seed text equality alone is not a stochastic correctness test.

For an API smoke test, both engines expose an OpenAI-compatible endpoint. Change the port and model to match the running server:

curl -sS http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"meta-llama/Llama-3.1-8B-Instruct","messages":[{"role":"user","content":"Explain why speculative decoding can reduce decode latency."}],"temperature":0,"max_tokens":128}'

A successful response proves basic serving, not that speculation is active or beneficial. Inspect startup logs and acceptance metrics. A long accepted prefix can still lose overall if a tree verifies many unused nodes or draft KV storage reduces the number of concurrent requests that fit.

11. Choosing a Starting Point

#

My engineering recommendation is to choose by the bottleneck and available checkpoints:

• A target with native MTP: establish a short-MTP baseline first; its integration and weight availability simplify comparison.
• A strong matched EAGLE-3 draft: test it for interactive latency, then verify that the advantage persists at service concurrency.
• Serial drafting dominates: evaluate DFlash with a workload-matched checkpoint and a modest supported block size.
• DFlash misses plausible alternative paths: DDTree is a useful research direction when extra verification capacity is available.
• Acceptance and load vary substantially: evaluate DSpark’s sequential correction and adaptive verification separately, then together.
• Existing Medusa weights: measure the actual engine proposer and acceptance policy rather than assuming the original paper’s behavior.

The deciding quantity is useful committed output per unit of total serving work. More heads, deeper drafts, wider trees, and longer blocks are only helpful when they improve that quantity under the application’s latency and throughput requirements.

References

#

• Foundations: Fast Inference from Transformers via Speculative Decoding.
• Heads and native prediction: Medusa, Multi-token Prediction, DeepSeek-V3.
• Feature drafting: EAGLE, EAGLE-2, EAGLE-3.
• Parallel and semi-autoregressive drafting: DFlash, DDTree, DSpark.
• Serving: vLLM speculative decoding, SGLang speculative decoding, DSpark in SGLang.
• Training and checkpoints: DeepSpec, DFlash implementation, DDTree implementation.