Qwen3.8-27B-DFlash2-W4A16-MLP-only

Qwen3.8-27B-DFlash2-W4A16-MLP-only

This is not a standalone language model. It is a DFlash2 speculative-decoding draft: a selective MLP-only W4A16 Compressed-Tensors quantization of z-lab/Qwen3.8-27B-DFlash2.

What is quantized

Exactly five draft MLP cores are selectively quantized: 15 source matrices (gate, up, and down across five MLPs), represented as 10 intended fused runtime quantized modules (gate_up_proj and down_proj for each MLP). Weights are INT4 with BF16 activations, group size 128, symmetric quantization, and no model zero points.

QKV, o_proj, the DFlash2 dynamic-convolution paths, shared FC, selector, norms, and auxiliary weights remain BF16.

vLLM compatibility and serving

The tested vLLM-XPU setup selectively used the draft packed-module-mapping fix from vLLM PR #53122 so the fused MLP projections are recognized correctly. QKV remains BF16 in this checkpoint, so the separate quantized-QKV part of that PR was not used. The PR was open when this model was published. Use a vLLM revision with equivalent draft packed-module mapping behavior.

The benchmark target was RedHatAI/Qwen3.8-27B-INT4, served under the alias Qwen3.8-27B. The tested setup used seven speculative tokens, probabilistic draft sampling, seed 42, and FP8 KV cache. It used the target repository's chat_template.jinja; place that file at ./chat_template.jinja or substitute its local path.

The following are my vllm serving flags during the testing phase.

vllm serve RedHatAI/Qwen3.8-27B-INT4 \
  --dtype bfloat16 \
  --served-model-name Qwen3.8-27B \
  --block-size 64 \
  --no-enable-prefix-caching \
  --seed 42 \
  --host 127.0.0.1 \
  --port 8080 \
  --gpu-memory-utilization 0.94 \
  --max-model-len 32768 \
  --kv-cache-dtype fp8 \
  --max-num-seqs 1 \
  --enable-chunked-prefill \
  --max-num-batched-tokens 16384 \
  --long-prefill-token-threshold 0 \
  --chat-template ./chat_template.jinja \
  --chat-template-content-format auto \
  --enable-auto-tool-choice \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_xml \
  --speculative-config '{"method":"dflash","model":"0xMiami305/Qwen3.8-27B-DFlash2-W4A16-MLP-only","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}' \
  --override-generation-config '{"temperature":1.0,"top_k":20,"top_p":0.95,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}' \
  --default-chat-template-kwargs '{"enable_thinking":true,"reasoning_effort":"medium","preserve_thinking":false}' \
  --language-model-only

On the tested vLLM-XPU stack, DFlash2 prefix caching had to be explicitly disabled; leaving the default enabled produced silent corruption in testing. The example therefore uses --no-enable-prefix-caching.

Single-run benchmark

In this single corrected-stack run, measured speculative acceptance was higher for the W4A16 draft at every position K1-K7. These are point estimates from one run per draft and should not be interpreted as a statistically established quality improvement. The largest observed point-estimate gains were in the later speculative positions, while aggregate raw acceptance was 0.754 percentage points higher. Observed throughput was 8.24% higher, and the profiled non-KV VRAM footprint was 1.67 GiB lower (about 6.7%).

Single run per draft; point estimate only, no statistical-significance claim. This was a fixed-order run on one Intel Arc Pro B70 with an identical 128-prompt GSM8K selection and order. Prefix caching was explicitly disabled. VRAM values are specific to the tested target and runtime configuration, not universal hardware-independent model properties.

Benchmark command

The benchmark used dflash 0.1.0; both compared runs used identical 128 prompt identities and order.

uvx dflash benchmark openai \
  --base-url http://127.0.0.1:8080 \
  --model Qwen3.8-27B \
  --dataset gsm8k \
  --num-prompts 128 \
  --concurrency 1 \
  --reasoning medium \
  --temperature 1 \
  --top-p 0.95 \
  --top-k 20 \
  --max-new-tokens 4096

License and attribution

License metadata is apache-2.0. The direct quantized source is z-lab/Qwen3.8-27B-DFlash2, whose model card contains the DFlash2 algorithm and paper citations and states that it mirrors incoai/Qwen3.8-27B-DFlash2. See THIRD_PARTY_NOTICES.md for detailed attribution and third-party notices.