Qwen3.8-27B-DFlash2-W4A16-g128-sym-GPTQ

Qwen3.8-27B DFlash2 MTP Draft Model — W4A16 Quantized

This repository contains a 4-bit weight-only quantized draft model for speculative decoding with Qwen3.8-27B. The drafter implements the DFlash2 (block diffusion) architecture, which predicts a block of tokens in a single forward pass and uses a lightweight path selector to trace a coherent sequence through the candidate tokens.

The drafter shares the target model's embed_tokens and lm_head; neither is shipped here, which keeps the footprint minimal.

Model Details

Quantized Components

All 35 linear layers across the five draft layers are quantized to INT4:

The following components are preserved in BF16:

Benchmark Results

Benchmarks were run on a single RTX 5090 (32 GB) with gratex/Qwen3.8-27B-W4A16-g128-sym-GPTQ as the target model. Prompts use unique random seeds to prevent prefix-cache hits from skewing the results. Each configuration was measured three times.

Throughput and Acceptance

Speedup vs No Speculative Decoding

Choosing num_speculative_tokens

The optimal number of speculative tokens depends on concurrency:

• Low concurrency (C=1, interactive) — num_speculative_tokens=7 gives the maximum speedup (2.9–3.0×).
• High concurrency (C=8, multi-user) — num_speculative_tokens=3 gives the maximum speedup (1.72–1.75×), because smaller draft blocks leave more KV cache for concurrent requests.
• Dynamic switching — vLLM supports num_speculative_tokens_per_batch_size to switch automatically based on batch size (see the vLLM speculative decoding docs).

FP8 KV Cache

Using --kv-cache-dtype fp8_e4m3 increases KV cache capacity by roughly 85% with no measurable impact on acceptance or throughput. This is recommended for multi-user scenarios.

Usage with vLLM

Prerequisites

• vLLM ≥ 0.26.1 with DFlash2 support (merged via PR #52816).
• The vllm/vllm-openai:cu129-nightly Docker image (built after 2026-08-21) includes this support.

Quick Start (Docker)

docker run -d --name vllm-dflash2 \
--runtime=nvidia --gpus '"device=0"' \
-p 8000:8000 \
-v /path/to/target-model:/workspace/base \
-v /path/to/this-model:/workspace/draft \
-v ./scripts/check_drafter.py:/tmp/check_drafter.py \
-v ./scripts/apply_dense_kv_fix.py:/tmp/apply_dense_kv_fix.py \
--ipc=host --shm-size=16g \
--entrypoint bash \
vllm/vllm-openai:cu129-nightly \
-c 'python3 /tmp/check_drafter.py /workspace/draft && \
python3 /tmp/apply_dense_kv_fix.py && \
exec vllm serve /workspace/base \
--speculative-config "{\"method\":\"dflash\",\"model\":\"/workspace/draft\",\"num_speculative_tokens\":3}" \
--trust-remote-code --dtype bfloat16 --max-model-len 8192 \
--kv-cache-dtype fp8_e4m3 --gpu-memory-utilization 0.98 \
--max-num-seqs 8 --max-num-batched-tokens 2048 \
--enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3'

Allow roughly 240 seconds for model loading and CUDA graph compilation, then verify with:

curl http://localhost:8000/health

The vLLM Patch (Required for Quantized Drafters)

Due to vLLM issue #51581, the DFlash2 loader in qwen3_dflash.py slices qkv_proj.weight directly, which fails when the drafter is quantized (packed weights have no .weight attribute, or .weight is packed as uint8/float8). A patch is therefore required.

The patch adds a _dense_kv_rows() helper that dequantizes the packed qkv_proj before slicing. It is included in this repository at scripts/apply_dense_kv_fix.py.

Run check_drafter.py and apply_dense_kv_fix.py before starting vLLM, as shown in the Quick Start above.

Scripts

All scripts live in scripts/.

apply_dense_kv_fix.py — vLLM Patch (v4)

Patches qwen3_dflash.py to fix bug #51581 for quantized DFlash2 drafter formats. It injects a _dense_kv_rows() helper that dequantizes the fused QKV projection before slicing, dispatching on the quantization format:

Every path either returns a valid tensor or raises ValueError with diagnostics. Formats not yet validated (for example ModelOpt NVFP4 via the weight_scale_2 marker, bitsandbytes, GGUF) fail loudly with a message that names the supported alternatives.

The patch was validated on hardware (RTX 5090, vLLM nightly) against three target quantizations:

Values are draft-token acceptance rates. All 12 combinations boot and draft correctly.

check_drafter.py — Pre-flight Completeness Check

A DFlash2 drafter that has lost candidate_selector, fc, or hidden_norm (for example through a naive transformers/llmcompressor re-quantization) loads in vLLM without error but yields 0% speculative acceptance. This script verifies that the checkpoint contains the required modules before vLLM starts:

python3 scripts/check_drafter.py /path/to/drafter
# Exit 0 = complete, 1 = incomplete (message printed), 2 = invalid path

quant_dflash2.py — RTN/GPTQ Quantization

The recommended way to quantize a drafter while preserving all DFlash2 modules. See "Reproducing the Quantization" below.

bench_dflash2_conc.py — Benchmark Harness

Measures throughput and acceptance at a given concurrency. See "Running Benchmarks".

Other Scripts

Note on transformers/llmcompressor: their quantizers load the drafter through transformers, which does not recognize the DFlash2 architecture and silently omits candidate_selector.*, fc, and hidden_norm from the output checkpoint, producing a drafter with 0% acceptance and no error. The manual scripts in this repository quantize the linear weights directly from the safetensors while preserving every other module.

Repository Contents

├── model.safetensors          # Quantized drafter weights (1.19 GB)
├── config.json                # Model config with quantization_config
├── README.md                  # This file
└── scripts/
    ├── apply_dense_kv_fix.py  # vLLM patch (v4, universal quantization support)
    ├── check_drafter.py       # Pre-flight completeness check
    ├── quant_dflash2.py       # RTN/GPTQ quantization script
    ├── gptq_utils.py          # GPTQ + RTN quantization primitives
    ├── quant_nvfp4.py         # NVFP4 quantization (preserves all modules)
    ├── quant_fp8.py           # FP8 quantization (preserves all modules)
    ├── bench_dflash2_conc.py  # Benchmark script (concurrency + acceptance)
    └── launch_vllm_dflash2.sh # Example Docker launch script

Reproducing the Quantization

# Install dependencies
pip install compressed-tensors safetensors torch

# Quantize (RTN — no calibration data required)
python scripts/quant_dflash2.py \
  incoai/Qwen3.8-27B-DFlash2 \
  ./output \
  --rtn --bits 4 --fc-bits 4 --group 128

Quantizing to FP8 / NVFP4 / MXFP4

The alternative-format scripts take an input drafter directory and an output directory:

# FP8 (E4M3, per-tensor)
python scripts/quant_fp8.py /path/to/bf16-drafter ./fp8-output

# NVFP4 (E2M1 + FP8 group scales, W4A16)
python scripts/quant_nvfp4.py /path/to/bf16-drafter ./nvfp4-output

# MXFP4 (E2M1 + E8M0 scales, W4A16)
python scripts/quant_mxfp4.py /path/to/bf16-drafter ./mxfp4-output

All three preserve candidate_selector, fc, hidden_norm, and the convolution kernels; only the linear projection weights are quantized. The output is a compressed-tensors checkpoint readable by the patched vLLM loader.

Running Benchmarks

# Start vLLM (see above), then:
python scripts/bench_dflash2_conc.py  # example: python scripts/bench_dflash2_conc.py 3 8

Troubleshooting

Acknowledgments

• incoai for the original Qwen3.8-27B-DFlash2 model.
• syv-ai for DFlash2 drafter calibration and quantization research.
• The vLLM team for DFlash2 integration (PR #52816).
• Gratex International, a.s. for providing the hardware and infrastructure used for quantization, benchmarking, and research.

License

Apache 2.0, following the original DFlash2 drafter license.