Qwen3.8-27B-DFlash2-INT8-LowLatency

Qwen3.8-27B-DFlash2-INT8-LowLatency

The official DFlash2 draft for Qwen3.8-27B (z-lab/Qwen3.8-27B-DFlash2, 5 Qwen3-style layers, 1.92 B parameters, 3.85 GB in BF16) with its MLP, o_proj and fc weights in INT8 (main, 2.30 GB) or INT4 (int4 revision, 1.51 GB), packed as compressed-tensors so vLLM's Marlin kernel reads fewer bytes per draft step.

Built as the companion of Twu31/Qwen3.8-27B-AWQ-INT4-MTP-LowLatency, an INT4 target for interactive, thinking-off serving. With that target on one RTX 6000 Ada, the BF16 draft already brings decode from 18.5 ms per token (no speculation) to 10.0 ms; every draft step also reads the whole 3.85 GB draft, which on a 960 GB/s card is about 4 ms of the step. This build cuts that read to 2.30 GB (INT8) or 1.51 GB (INT4) and the draft's GPU footprint by 1.4 GB (INT8) or 2.2 GB (INT4).

TL;DR · same architecture and config as the z-lab draft, weights post-hoc RTN g128 symmetric, no calibration data · loads in vLLM 0.29 through 0.33 with "method":"dflash" against BF16 or quantized targets · q/k/v projections, the grouped-convolution kernel projections and the candidate selector stay BF16 (see Why q/k/v stay BF16) · with the INT4 target on one RTX 6000 Ada a decode step takes 25.3 ms with the INT8 draft and 24.3 ms with INT4, against 27.1 ms with the BF16 draft, the same on vLLM 0.31, 0.32 and 0.33; on short chat replies that is 9.3 / 9.0 against 10.2 ms per token, with 2.7 to 2.8 tokens accepted per step either way (Measured) · vLLM 0.33 removes most of DFlash2's TTFT penalty on this hybrid target: 80 / 79 ms TTFT and 235 / 228 ms per 18-token turn with the INT8 / INT4 draft, against about 240 ms and 385 to 405 ms on 0.32 · speculation only changes speed: at temperature 0 the replies match running the target alone except on 1 or 2 of our 18 prompts, the same run-to-run variation the target shows between two restarts.

Use

vllm serve Twu31/Qwen3.8-27B-AWQ-INT4-MTP-LowLatency \
  --served-model-name qwen38 --max-model-len 12288 --max-num-seqs 8 \
  --enable-prefix-caching --mamba-ssm-cache-dtype bfloat16 \
  --reasoning-parser qwen3 --default-chat-template-kwargs '{"enable_thinking": false}' \
  --speculative-config '{"method":"dflash","model":"Twu31/Qwen3.8-27B-DFlash2-INT8-LowLatency","num_speculative_tokens":7}'

• "revision":"int4" inside the speculative config selects the INT4 build.
• Works with any Qwen3.8-27B target that vLLM can serve (BF16, FP8, INT4); the draft shares the target's embeddings and lm_head, so a quantized target lm_head is fine.
• Needs vLLM 0.29 or newer (DFlash2 support, vllm#52816); tested on 0.29.0 through 0.33.0.
• Check it is accepting: curl :8000/metrics | grep spec_decode; on chat traffic expect about 0.25 accepted per drafted token at K=7, that is 2.5 to 2.8 tokens per step.
• K=7 (the draft's block size minus one) was the best setting on short replies; the extra verification is nearly free at batch 1.

Measured

INT4 target (main of the target repo), one RTX 6000 Ada 48 GB, conversational prompts (≈1.5 k-token system prompt + 2 turns of history + a user utterance), greedy, thinking off, 18 prompts × 3 runs, medians, KV cache fixed at 10 GiB. The 0.31.0 / 0.32.0 rows were run back to back on 2026-10-23, each 0.31.0 arm right before its 0.32.0 arm, and the 0.32.0 / 0.33.0 rows the same way on 2026-11-06 (the BF16-draft row of that pair with --prefix-match-unit 16 added). Medians over these 18-token replies move by up to 0.5 ms/tok from run to run; Per decode step below has the steadier measurement.

Both quantized drafts keep the BF16 draft's acceptance (2.72 to 2.79 tokens per step is within run-to-run noise). In the medians above INT8 and INT4 often land on the same ms/tok, but timed per step (below) INT4 is about 1 ms faster than INT8, so the draft read is still part of every step. main is INT8 because its weights are within 0.7 % of BF16; int4 (12 % mean relative weight error from plain rounding) is published because its acceptance measured the same, not because it is expected to generalise to every workload.

On vLLM 0.33.0 all three drafts get the same TTFT drop, from about 240 to 80 to 85 ms: 0.33 no longer gives back the last cached block on a prefix hit for DFlash (vllm#54163); more on that below. Decode per step is unchanged.

Per decode step

A reply of about 18 tokens is only six or seven speculative steps, so its ms/tok moves by about 1.5 ms with each step more or less, and the median of 54 such values can sit in the gap between two clusters. A re-test on 2026-10-23 therefore also timed fixed-length replies (6 of the prompts, 192 tokens each with ignore_eos, time between the first and last streamed token divided by the number of steps) and profiled the steps with Nsight Systems, each draft on 0.31.0 and 0.32.0 in turn:

On 0.33.0 (2026-11-06, the same fixed-length timing, no profiler) a step takes 26.9 ms with the BF16 draft, 25.1 ms with INT8 and 24.1 ms with INT4, against 27.1 / 25.1 / 24.2 ms for a 0.32.0 run right before it, so the 7 % / 10 % saving holds.

The target's verification pass takes 18.7 ms of every step in every row and the GPU is idle for about 0.1 ms per step, so whatever the draft saves comes straight off the step: 1.9 ms (7 %) with INT8 and 2.8 ms (10 %) with INT4, on both versions. The draft forward includes 1.5 ms for scoring candidates with the target's lm_head, the same for all three drafts.

An earlier version of this card said vLLM 0.32.0 removed the speed gain, from the 9.0 vs 9.4 ms/tok medians in the last rows of the table above. That was the median effect: on requests whose reply text was identical on both versions, the median difference in decode time was 1 ms or less, out of 170 to 200 ms per reply. The 2.70 tokens per step for INT4 on 0.32.0 came from one compiled build; a fresh 0.32.0 compile gives 2.79 (0.31.0: 2.78). On every version the quantized drafts also save 1.4 GB (INT8) or 2.2 GB (INT4) of GPU memory.

TTFT with any DFlash2 draft is higher than with the target alone or with MTP on a hybrid GDN target: the draft's sliding-window layers switch off vLLM's fine-grained prefix-cache hits for the whole model (vllm#53786, vllm#57032, both still open), so hits come in whole 448-token blocks. Up to 0.32 vLLM also dropped the last of those blocks on every hit, about 240 ms TTFT on a 1.5 k-token prompt; 0.33 keeps it, about 80 ms. On 0.33 an 18-token turn with the INT4 draft (228 ms) is level with the target repo's MTP recipe with --prefix-match-unit 16 (235 ms), but later turns of a conversation still climb to 90 to 130 ms TTFT on the block-granular hits (MTP: 77 to 79 ms), and DFlash2 stays a single-stream choice (597 ms per turn at five streams against 287 ms with MTP). So for short-reply chat MTP is still the default, and DFlash2 pays off on longer replies. Details on the target's card.

What is quantized, what is not

Sizes on disk: BF16 3.85 GB · INT8 2.30 GB · INT4 1.51 GB.

Why q/k/v stay BF16

vLLM builds the draft's fused context-KV projection at load time from the raw qkv_proj.weight. A draft whose q/k/v projections are quantized has weight_packed instead and fails to load on vLLM 0.29 and 0.30 with 'QKVParallelLinear' object has no attribute 'weight' (vllm#51581). vLLM 0.31.0 and 0.32.0 run a quantized qkv_proj through its quant method instead, and drafts with INT8 or INT4 q/k/v load there with the same acceptance (2.75 to 2.78 tokens per step at K=7 with the INT4 target). 0.33.0 loads them too, but in our run they accepted 2.66 (INT8) and 2.62 (INT4) per step, while the two drafts here held 2.74 / 2.76. Keeping the three projections in BF16 costs 0.3 GB and works on every version from 0.29, which is why both revisions here keep them.

One detail for anyone building their own: the ignore list has to name the fused module as vLLM sees it, re:.*self_attn\.qkv_proj$, in addition to the per-projection names. Draft models do not receive vLLM's packed_modules_mapping (vllm#53116), so an ignore that only lists q_proj / k_proj / v_proj never matches and the fused layer is built quantized, which then fails with 'QKVParallelLinear' object has no attribute 'data'. The config.json here carries both.

How it was made

Post-hoc round-to-nearest, symmetric, group size 128, using compressed-tensors' own quantize and pack routines so the bit layout is the one vLLM expects (pack-quantized, config_groups.group_0 targeting Linear, the BF16 modules listed in ignore). No calibration data: the draft's inputs are hidden states of the target, not tokens, so the usual calibration pipelines do not apply without capturing those activations. INT8 RTN is lossless for practical purposes (mean relative weight error 0.7 %); INT4 RTN is coarser (12 %) and is published only because it was measured. For a calibrated INT4 draft, syvai/Qwen3.8-27B-DFlash2-W4A16 quantized the same draft with GPTQ on captured drafter inputs and reports acceptance on par with BF16; that build quantizes q/k/v too and therefore needs a patched vLLM.

The conversion is a short script over those routines: it reads the z-lab safetensors, quantizes the listed matrices, writes one model.safetensors plus a config.json that adds quantization_config and leaves every other field as z-lab shipped it.

Revisions

Limitations

• Measured only with the INT4 target above, on short Chinese and English chat replies, single stream. Long-context (the draft's window is 2048 tokens) and long-reply acceptance were not measured here; z-lab's numbers for the BF16 draft are the reference for those.
• The draft carries max_position_embeddings 262144 like the target; if you pair it with a target served at a longer context on vLLM 0.32 or older, see vllm#57941; 0.33 raises the draft's limit to the target's context (vllm#63251).
• Same fine-grained prefix-cache limitation as the BF16 draft (above).

Acknowledgements

z-lab / Inco for DFlash2 and the draft; Qwen team for Qwen3.8; the vLLM and compressed-tensors maintainers.