DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation

In speculative decoding, verifying a few extra draft tokens may help one request, but under heavy load those tokens can consume GPU capacity that could have served other requests. DSpark (Xin Cheng et al., DeepSeek-AI and Peking University) solves this by estimating the chance that each draft token will be accepted and checking how busy the GPU is. When the GPU has capacity, it verifies more tokens. If not, it verifies fewer, because extra tokens would slow the whole batch. As a result, this method helps keep total throughput high.

TL;DR

DSpark combines a parallel drafting backbone with a tiny sequential Markov head to reduce suffix decay without using a full autoregressive drafter. Then it uses calibrated token confidence and profiled GPU step times to choose how many draft tokens each request should verify under the current load. The authors report higher accepted lengths across four target models and substantial production throughput gains over MTP-1.

1. Why decoding is the bottleneck

LLM inference has two phases, prefill and decode. In prefill, shown at the top of the figure, the model processes the whole prompt in one forward pass, building the KV cache and producing the first output token. Every prompt token is processed in parallel, so each weight read from memory is reused across many tokens. For sufficiently large prompts, prefill is generally compute-bound. It spends most of its time computing rather than waiting for data.

Decode, shown at the bottom of the figure, produces the remaining tokens one step at a time until the model emits an end-of-sequence token or reaches the output’s maximum length. Each token depends on the previous one, so every step reads the model weights and KV cache from GPU memory to produce the next token. At small batches, there is little computation for each byte read, and the KV-cache reads grow with the output, so the GPU spends much of its time waiting for data. As a result, this phase usually dominates end-to-end latency.

There are many ways to tackle this problem, including prefill/decode disaggregation (so long prefills don’t stall decoding), quantization (fewer bytes to read), hardware with more memory bandwidth (such as NVIDIA Rubin), and speculative decoding (more tokens per read). This article focuses on speculative decoding.

2. What is speculative decoding?

Speculative decoding takes advantage of the underutilized computing power during decoding. Modern GPUs have plenty of compute, so a large model can verify several tokens from a small model in one forward pass for nearly the same cost as generating a single token. A smaller drafter proposes the next k tokens, called a draft block. The large model, called the target model, then checks them all in one pass. If a draft token is rejected, the system keeps the accepted prefix and uses a token from the large model instead.

The speedup depends on two things: how cheap the draft is and how many draft tokens get accepted. The two main types of drafters trade these off differently.

2.1 Autoregressive drafter

The drafter generates the k tokens one at a time, each conditioned on the previous one. Drafts are high quality because every token sees the tokens before it, but drafting cost grows linearly with k, so draft lengths stay short. Popular methods include EAGLE-3 and MTP, often paired with tree-based verification to make up for the short drafts.

2.2 Parallel drafter

The drafter predicts all k tokens in a single forward pass. Drafting cost stays roughly constant as k grows, but each position is predicted without seeing the actual tokens before it, so the acceptance rate drops at later positions in the block. Popular methods include Medusa, PARD, DART, and DFlash.

3. What’s wrong with the prior methods?

3.1 Autoregressive drafters

As shown on the left of the figure, each draft token needs its own drafter forward pass, so drafting latency grows linearly with the block length. To keep that latency low, these drafters use short blocks and shallow networks, then use tree-based verification to compensate for the short blocks by verifying several candidate paths at once. This sends more tokens to the target model and can reduce aggregate serving throughput under concurrency.

A shallow network is also less accurate, and this matters most at the first draft position. The target only accepts a prefix, so if the first draft token is wrong, the whole block is thrown away. As shown on the right, EAGLE-3 is accepted less often at the first position than DFlash, a parallel drafter that can afford a much deeper network, with 0.81 vs. 0.88 on math and 0.53 vs. 0.72 on chat.

3.2 Parallel drafters

A parallel drafter predicts every position in the block in one forward pass, so no position can see which token was actually sampled before it. Instead, each position must account for all the tokens that might come before it. As a result, when the context allows for several continuations, different positions can lock onto different ones.

For example, as shown on the left of the figure, if both “of course” and “no problem” fit, a parallel drafter may propose “of problem” or “no course”. The paper calls this issue a multi-modal collision, and it causes suffix decay. The chance that a token is accepted tends to fall at later positions, even when every earlier token in the block was accepted. Therefore, for a pure parallel drafter, each extra position in a longer block tends to be worth less than the one before.

The right panel plots conditional acceptance at each position, that is, how often a token is accepted given that every token before it was accepted. DFlash starts high but falls along the block, from 0.87 to 0.78 on code and from 0.72 to 0.63 on chat. However, EAGLE-3 does the opposite. It starts lower (0.53 on chat, the same first-position number as in 3.1), but because each of its tokens sees the ones before it, it holds steady or even improves deeper into the block (up to 0.74 on chat).

3.3 Verification length

Parallel drafting makes long blocks cheap to produce, but verifying every drafted token is not cheap. The ideal verification length changes along two axes.

• Data: on the left of the figure, acceptance depends on the request. Structured text, such as code, sustains much higher acceptance than open-ended chat, so any single length is too short for some requests and too long for others.
• System load: on the right of the figure, verifying an extra token at light load can be nearly free when the GPU has idle capacity and is waiting on memory anyway, so even a rejected token costs almost nothing. Under heavy load, every token sent for verification adds to the batch that the target model processes in that step, and the step gets slower as that batch grows. A token that is likely to be rejected takes batch capacity that could have served another request.

Put together, the two methods have opposite disadvantages. Autoregressive drafters are weak at the first position, and parallel drafters become weaker at later ones. The first position matters most since a miss there throws away the whole block. DSpark targets this gap by retaining the parallel drafter’s strong first position while reducing decay at later positions.

Furthermore, prior methods either verify a fixed number of tokens or use a static confidence threshold. Neither accounts for the current system load. That can work under single-request assumptions, but not in a high-concurrency production system. Before DSpark, DeepSeek’s production serving used MTP-1, a single draft token, because deploying a static multi-token drafter (MTP-3 or MTP-5) degraded aggregate throughput under high concurrency.

4. How DSpark works

DSpark addresses the problems from Section 3 with two parts. The first is semi-autoregressive generation, which keeps the strong first position of a parallel drafter and reduces suffix decay after it. The second is confidence-scheduled verification, which decides how many draft tokens are worth sending to the target based on how confident the drafter is and how busy the GPU is.

4.1 Semi-autoregressive generation

DSpark splits drafting into two stages, as shown in the figure. In the parallel stage, a heavy backbone (DFlash in the paper) runs one forward pass over the whole block and gives a base guess for every position. This is where most of the drafting work happens, so drafting time stays roughly constant as the block grows. Then, in the sequential stage, a tiny Markov head walks through the block from left to right. At each position, it adds a bias to the base guess that depends on the token just sampled, then samples the next token.

For example, if both “of course” and “no problem” fit, the backbone alone guesses “of / no” and “course / problem”. Once position 1 samples “of”, the Markov head boosts “course” and suppresses “problem” at position 2, so the pair stays coherent. In short, this acts like a parallel drafter for speed and uses a small sequential head to keep adjacent words coherent without sacrificing that speed.

The Markov head has to be tiny because it runs once per position, one after another, using a low-rank lookup table (rank 256 by default). The paper also tests an RNN head that remembers the whole prefix instead of only the previous token, but it only gives small extra gains.

4.2 Confidence-scheduled verification

Even with better drafts, verifying every drafted token is wasteful because the last few tokens of a block are often rejected. DSpark adds two pieces to decide how much of each block to verify. A confidence head gives each draft token a confidence score, and a hardware-aware prefix scheduler uses those scores together with the current GPU load to choose how many tokens from each request to send to the target.

In the figure, the scheduler keeps “jumps over a” and drops “lazy”, so the target verifies only three tokens. The target then works like normal speculative decoding. It accepts “jumps over”, rejects “a”, and adds its own token “the”. The paper shows that this scheduling keeps the output distribution exactly the same as the target model’s.

4.2.1 Confidence Head

As shown in Step 1 of the figure, the confidence head is a small linear layer with a sigmoid on top. For each draft position k, it looks at the backbone’s hidden state and the embedding of the previous draft token, and outputs a score c_k. This is the chance that token k is accepted, given that every token before it was accepted. During training, it learns to match the real acceptance rate, which the paper computes from how far the draft distribution is from the target’s.

Raw neural network scores are usually overconfident, so DSpark calibrates them first. As shown in Step 2, it uses Sequential Temperature Scaling (STS), fitting one temperature per position from left to right on held-out data. This rescales the scores to match real acceptance rates without changing their order, reducing the calibration error from 3–8% to about 1%. Each calibrated c_k still covers only one position, while the target model accepts draft tokens as a continuous prefix. In Step 3, DSpark multiplies the scores together into a prefix survival probability, c_1 × c_2 × … × c_k, which gives the chance that every token up to position k is accepted and never increases along the block.

4.2.2 Hardware-Aware Prefix Scheduler

With the survival scores ready, the scheduler decides how many draft tokens the whole batch should verify. Each extra token can add accepted output, but it also makes the target’s step slower. The scheduler ranks every request’s draft tokens by survival and adds them one at a time. Survival never rises along a draft, so each request keeps a prefix. After each addition, it looks up the step time in a table measured at startup and stops once the expected accepted tokens per second drops, which keeps the output exact. In the figure, the scheduler keeps 13 of the 15 tokens under light load but only 9 under heavy load.

Moreover, a production engine needs two small adjustments, shown in the figure above. On the left, GPU step time rises in jumps rather than smoothly, so expected throughput can dip before its true peak. The scheduler therefore searches the whole table instead of stopping at the first dip, which costs nothing since every entry was measured at startup. On the right, the engine prepares the next batch while the GPU is still busy, so DSpark sets the batch size from scores two steps old and picks the tokens with the latest scores. Since those older scores were fixed before the current tokens existed, the full search still keeps the output exact.

5. Results

All results in this section are reported by the DSpark paper. I have not reproduced them.

5.1 Experimental Setup

All three drafters are trained with the same code and data in DeepSpec. The confidence scheduler is turned off for these offline tests so every drafter proposes a fixed 7-token block, which isolates draft quality from scheduling.

5.2 Experimental Results

Accepted length per round, from the paper’s Table 1. LCB is LiveCodeBench.

DSpark reaches the longest accepted length on every benchmark and every target model. On Qwen3-4B, 8B, and 14B, it accepts 30.9%, 26.7%, and 30.0% more tokens per round than EAGLE-3, and 16.3%, 18.4%, and 18.3% more than DFlash. The same lead holds on Gemma4-12B, showing that the design carries over to another model family.

Two patterns in the table stand out. First, acceptance varies sharply by task. On Qwen3-4B, DSpark keeps 5.57 tokens per round on math and 5.12 on code, compared with 3.49 on chat. That gap is why a single fixed verification length wastes compute on hard prompts and underuses easy ones. Second, EAGLE-3 beats DFlash across every Gemma4-12B benchmark, so a pure parallel drafter does not always beat an autoregressive one, yet DSpark still comes out ahead of both.

5.3 Experimental Analysis

1. Position-wise acceptance confirms the design

Tracking acceptance at each draft position backs up the behavior in Sections 3.1 and 3.2. DSpark keeps the deep parallel backbone’s strong first token (0.93 acceptance on math at position 1), and the Markov head keeps acceptance high and steady across the remaining positions instead of decaying.

2. A little autoregression goes a long way

• Depth: with a 7-token block, DSpark improves with each added layer from 1 to 5. The biggest jump happens between 1 and 2 layers, and a 2-layer DSpark already beats the 5-layer DFlash in every domain.
• Block length: DSpark beats DFlash at every block length tested, and the gap widens as blocks get longer. At 7 draft tokens, DSpark accepts 16% more on math, 15% more on code, and 18% more on chat. At 15 draft tokens, those gains grow to 30%, 26%, and 22%. Replacing the Markov head with an RNN head adds only small extra gains on longer blocks.
• Cost: at batch size 128, DSpark’s sequential head makes each round only 0.2% to 1.3% slower than DFlash across draft lengths from 4 to 16 tokens, while accepting up to 30% more tokens.

3. Verify smarter, not longer: the role of confidence scheduling

Before adding the full scheduler, the paper tests the confidence head on its own by dropping draft tokens below a fixed confidence cutoff. Raising that cutoff filters out tokens that would mostly be rejected anyway, pushing the acceptance rate of the kept tokens from 45.7% to 95.7% on chat, from 76.9% to 92.5% on math, and from 67.6% to 92.0% on code. Chat benefits the most because its later draft tokens miss more often. Still, a high acceptance rate on kept tokens does not guarantee a wall-clock speedup, since cutting too many tokens shrinks each round. A fixed cutoff also ignores GPU load, which is why production serving pairs the calibrated confidence head with the hardware-aware scheduler.

4. Production results

• Scalability: the production drafter uses three MoE layers with a sliding window attention of 128 plus the Markov head. To keep training fast on long contexts, workers send hidden states instead of full-vocabulary logits and pack short training blocks around sampled anchor positions.
• Hardware-aware prefix scheduler: the two production tweaks from Section 4.2.2 let the scheduler run alongside zero-overhead scheduling and CUDA graphs without changing the output distribution.
• High throughput with low latency: in DeepSeek’s deployment, KV-cache limits and live traffic keep batches below GPU compute saturation, so higher total throughput and faster per-user speed improve together. To verify a different number of tokens per request in the same batch, the engine flattens all tokens and tracks their request IDs, requiring changes to only two kernels (index-attention and compress).
• Results from real traffic: at moderate load (under 200 concurrent requests on V4-Flash and 150 on V4-Pro), the scheduler verifies about 4–6 tokens per request (including the target’s own token) compared with MTP-1’s fixed 2, and dials that budget down as load rises. DSpark replaced MTP-1 in production two weeks after the DeepSeek-V4 preview launch.

From the paper’s Sec. 5.4. At the stricter speed targets, MTP-1 can only serve a very small batch, so the paper reads the +661% and +406% as DSpark extending what is reachable, not as typical speedups.

5.4 Limitations and Future Work

One limitation noted in the paper is that the scheduler only saves work on the target model. The drafter still runs its full parallel backbone over the entire block on every step, even on hard requests where the scheduler ends up dropping most of the draft tokens. The authors suggest adding difficulty-aware early exit inside the drafter so hard prompts can skip generating the full block.

From my perspective, a few more trade-offs are worth keeping in mind.

• Offline tables measure draft quality, not speed: Table 1 runs with the confidence scheduler turned off and a fixed block size of 7. That is a clean way to compare drafters, but real end-to-end speedup still depends on drafter overhead, batch size, and target step time.
• Using scores from two steps ago can lag sudden changes: in the paper’s design, the batch size is picked from scores two steps earlier. That works well when acceptance stays steady from round to round, but if a request suddenly switches from predictable code to open-ended writing, the number of tokens the scheduler verifies takes two rounds to catch up.

6. Demo

vLLM already supports both parts of DSpark. Setting "method": "dspark" runs the semi-autoregressive drafter from Section 4.1, and setting "enable_adaptive_verification": true adds the confidence-scheduled verification from Section 4.2. With adaptive verification on, vLLM scores each draft token with the checkpoint’s confidence head and uses a step-time table profiled at startup to decide how many tokens each request should verify.

Each step below shows the commands for your own GPU machine. To run the same step on a Modal H100 instead, open Run it on Modal under it.

1. Install vLLM

First, we need to install vLLM in a Python environment, and in this demo we will use the bench extra to demonstrate DSpark. Then run vllm download-kernels to install FlashInfer’s precompiled kernels to reduce startup time. Even with Triton attention, vLLM uses FlashInfer for some operations.

pip install -U "vllm[bench]"
vllm download-kernels

pip install modal
modal setup

cat > dspark_modal.py <<'EOF'
import os
import signal
import subprocess
import time
import urllib.request

import modal

app = modal.App("dspark-demo")

# Step 1 on Modal: a container image with vLLM installed
image = (
    modal.Image.from_registry("nvidia/cuda:13.0.3-devel-ubuntu24.04", add_python="3.12")
    .entrypoint([])
    .apt_install("curl")
    .uv_pip_install("vllm[bench]")
    .run_commands("vllm download-kernels", gpu="H100")  # vllm needs a GPU to start
)

# Keep downloaded models and compiled kernels between runs
volumes = {
    "/root/.cache/huggingface": modal.Volume.from_name("dspark-hf-cache", create_if_missing=True),
    "/root/.cache/vllm": modal.Volume.from_name("dspark-vllm-cache", create_if_missing=True),
}


@app.function(image=image, gpu="H100!", volumes=volumes, timeout=60 * 60)
def run(server: str, client: str):
    # Terminal 1: start the server in the background
    server_process = subprocess.Popen(["bash", "-c", server], start_new_session=True)

    # Wait until the server answers, which happens after "Application startup complete"
    while True:
        try:
            urllib.request.urlopen("http://localhost:8000/health")
            break
        except OSError:
            if server_process.poll() is not None:
                raise RuntimeError("vllm serve exited before it was ready")
            time.sleep(5)

    # Terminal 2: run the client commands against the server
    subprocess.run(["bash", "-c", client], check=True)

    # Wait for the server's next stats log, then stop it like Ctrl+C
    time.sleep(15)
    os.killpg(server_process.pid, signal.SIGINT)
    server_process.wait()


@app.local_entrypoint()
def main(server: str, client: str):
    with open(server) as server_file, open(client) as client_file:
        run.remote(server_file.read(), client_file.read())
EOF

2. Launch vLLM with DSpark

Use --speculative-config to configure a DSpark drafter that matches your target model. The released Qwen3 drafters were trained with thinking mode turned off, so the server turns it off too. Prefix caching is disabled so that repeated benchmark runs do not reuse cached prompts and inflate the results. Adaptive verification also needs an attention backend whose CUDA graphs can handle a different number of tokens for each request. On GPUs such as the A100, L40S, and RTX PRO 6000, vLLM’s default FlashAttention 2 backend cannot do this and fails at startup, so the server uses Triton attention instead.

# Terminal 1: start the server
vllm serve Qwen/Qwen3-8B \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.85 \
  --no-enable-prefix-caching \
  --attention-backend TRITON_ATTN \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --speculative-config '{
    "method": "dspark",
    "model": "deepseek-ai/dspark_qwen3_8b_block7",
    "num_speculative_tokens": 7,
    "enable_adaptive_verification": true
  }'

Once the log prints Application startup complete, send a test request from a second terminal.

# Terminal 2: send a test request
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-8B",
    "messages": [
      {"role": "user", "content": "Write a Python function for binary search."}
    ],
    "max_tokens": 256,
    "temperature": 0
  }'

# Save the server command
cat > server.sh <<'EOF'
vllm serve Qwen/Qwen3-8B \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.85 \
  --no-enable-prefix-caching \
  --attention-backend TRITON_ATTN \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --speculative-config '{
    "method": "dspark",
    "model": "deepseek-ai/dspark_qwen3_8b_block7",
    "num_speculative_tokens": 7,
    "enable_adaptive_verification": true
  }'
EOF

# Save the test request
cat > client.sh <<'EOF'
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-8B",
    "messages": [
      {"role": "user", "content": "Write a Python function for binary search."}
    ],
    "max_tokens": 256,
    "temperature": 0
  }'
EOF

# Run both on a Modal H100
modal run dspark_modal.py --server server.sh --client client.sh

The reply should look like any normal Qwen3-8B answer. In the server log, a line starting with SpecDecoding metrics reports the mean acceptance length, the average number of tokens each round produces, including the target model’s own token.

3. Benchmark light and heavy load

In this demo, we will compare light load, which sends one request at a time, with heavy load, which keeps 64 requests in flight, across three setups: no speculative decoding, speculative decoding with EAGLE-3, and speculative decoding with DSpark.

3.1 Prepare the benchmark scripts

Save each load as a script to reuse with every server.

# Light load, one request at a time
cat > light.sh <<'EOF'
vllm bench serve \
  --backend openai-chat --endpoint /v1/chat/completions \
  --model Qwen/Qwen3-8B --skip-chat-template \
  --dataset-name hf --dataset-path philschmid/mt-bench \
  --num-prompts 80 --max-concurrency 1
EOF

# Heavy load, 64 requests at a time
cat > heavy.sh <<'EOF'
vllm bench serve \
  --backend openai-chat --endpoint /v1/chat/completions \
  --model Qwen/Qwen3-8B --skip-chat-template \
  --dataset-name hf --dataset-path philschmid/mt-bench \
  --num-prompts 320 --max-concurrency 64
EOF

3.2 No speculative decoding

This is the baseline, where the target model generates one token per step.

# Terminal 1: start the server
vllm serve Qwen/Qwen3-8B \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.85 \
  --no-enable-prefix-caching \
  --attention-backend TRITON_ATTN \
  --default-chat-template-kwargs '{"enable_thinking": false}'

# Terminal 2: light load
bash light.sh

# Terminal 2: heavy load
bash heavy.sh

# Save the server command
cat > server.sh <<'EOF'
vllm serve Qwen/Qwen3-8B \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.85 \
  --no-enable-prefix-caching \
  --attention-backend TRITON_ATTN \
  --default-chat-template-kwargs '{"enable_thinking": false}'
EOF

# Light load on a Modal H100
modal run dspark_modal.py --server server.sh --client light.sh

# Heavy load on a Modal H100
modal run dspark_modal.py --server server.sh --client heavy.sh

3.3 Speculative decoding with EAGLE-3

EAGLE-3 drafts one token at a time, so each extra draft token costs another drafter pass. Red Hat’s EAGLE-3 drafter for Qwen3-8B was trained with thinking turned off, and its model card runs it with three draft tokens. Its training data differs from DeepSeek’s drafters, so this compares the released drafters rather than reproducing the paper’s Table 1.

# Terminal 1: start the server
vllm serve Qwen/Qwen3-8B \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.85 \
  --no-enable-prefix-caching \
  --attention-backend TRITON_ATTN \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --speculative-config '{
    "method": "eagle3",
    "model": "RedHatAI/Qwen3-8B-speculator.eagle3",
    "num_speculative_tokens": 3
  }'

# Terminal 2: light load
bash light.sh

# Terminal 2: heavy load
bash heavy.sh

# Save the server command
cat > server.sh <<'EOF'
vllm serve Qwen/Qwen3-8B \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.85 \
  --no-enable-prefix-caching \
  --attention-backend TRITON_ATTN \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --speculative-config '{
    "method": "eagle3",
    "model": "RedHatAI/Qwen3-8B-speculator.eagle3",
    "num_speculative_tokens": 3
  }'
EOF

# Light load on a Modal H100
modal run dspark_modal.py --server server.sh --client light.sh

# Heavy load on a Modal H100
modal run dspark_modal.py --server server.sh --client heavy.sh

3.4 Speculative decoding with DSpark

3.4.1 With a fixed draft length

This uses the same DSpark drafter as Step 2 with adaptive verification turned off, so every request verifies all 7 draft tokens, just like standard speculative decoding.

# Terminal 1: start the server
vllm serve Qwen/Qwen3-8B \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.85 \
  --no-enable-prefix-caching \
  --attention-backend TRITON_ATTN \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --speculative-config '{
    "method": "dspark",
    "model": "deepseek-ai/dspark_qwen3_8b_block7",
    "num_speculative_tokens": 7,
    "enable_adaptive_verification": false
  }'

# Terminal 2: light load
bash light.sh

# Terminal 2: heavy load
bash heavy.sh

# Save the server command
cat > server.sh <<'EOF'
vllm serve Qwen/Qwen3-8B \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.85 \
  --no-enable-prefix-caching \
  --attention-backend TRITON_ATTN \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --speculative-config '{
    "method": "dspark",
    "model": "deepseek-ai/dspark_qwen3_8b_block7",
    "num_speculative_tokens": 7,
    "enable_adaptive_verification": false
  }'
EOF

# Light load on a Modal H100
modal run dspark_modal.py --server server.sh --client light.sh

# Heavy load on a Modal H100
modal run dspark_modal.py --server server.sh --client heavy.sh

3.4.2 With adaptive verification

This is the same server as Step 2, with both the drafter and adaptive verification turned on.

# Terminal 1: start the server
vllm serve Qwen/Qwen3-8B \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.85 \
  --no-enable-prefix-caching \
  --attention-backend TRITON_ATTN \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --speculative-config '{
    "method": "dspark",
    "model": "deepseek-ai/dspark_qwen3_8b_block7",
    "num_speculative_tokens": 7,
    "enable_adaptive_verification": true
  }'

# Terminal 2: light load
bash light.sh

# Terminal 2: heavy load
bash heavy.sh

# Save the server command
cat > server.sh <<'EOF'
vllm serve Qwen/Qwen3-8B \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.85 \
  --no-enable-prefix-caching \
  --attention-backend TRITON_ATTN \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --speculative-config '{
    "method": "dspark",
    "model": "deepseek-ai/dspark_qwen3_8b_block7",
    "num_speculative_tokens": 7,
    "enable_adaptive_verification": true
  }'
EOF

# Light load on a Modal H100
modal run dspark_modal.py --server server.sh --client light.sh

# Heavy load on a Modal H100
modal run dspark_modal.py --server server.sh --client heavy.sh

3.5 Results

The table below collects the heavy-load results from the screenshots above, all run on a Modal H100.

Under heavy load, these results line up with the paper in two ways.

• Acceptance length: with a fixed draft length, DSpark accepts 3.78 tokens per round on MT-Bench, close to the 3.72 the paper reports for Qwen3-8B in Table 1.
• Adaptive verification: it accepts fewer tokens per round, 3.57 against 3.78, yet gives the highest throughput and less than half the mean TTFT of the fixed length, as verifying fewer tokens protects the batch.

7. Takeaways

• Drafting and verification solve different problems. A deep parallel backbone with a tiny Markov head makes it cheap to draft long, coherent blocks, while a separate scheduler decides how many of those tokens are worth sending to the target model.
• Accepted tokens per round is not the only metric that matters. In production, the real goal is total system throughput within your latency target, which depends on batch load as much as draft accuracy.
• Draft length works best as a live scheduling choice. Checking more draft tokens pays off when the GPU has spare room, and checking fewer protects the batch when traffic is heavy.

Paper: Xin Cheng, Xingkai Yu, Chenze Shao, Jiashi Li, Yunfan Xiong and 28 co-authors (DeepSeek-AI and Peking University), “DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation”, arXiv:2607.05147, July 2026. All figures except the demo screenshots are illustrative redraws; paper-specific examples and numerical results are based on the DSpark paper.