Qwen3.8-27B-Dflash

Qwen3.8-27B DFlash Draft Model

A DFlash speculative-decoding draft model for Qwen/Qwen3.8-27B.

This is not a standalone chat model. It must be served alongside the Qwen3.8-27B target model, where it proposes tokens that the target model verifies through DFlash speculative decoding.

Early benchmark results

These are early results from a small benchmark and should be treated as an initial baseline, not a final performance claim. Results depend heavily on workload, hardware, inference engine, draft-token configuration, and token acceptance rate.

On the same benchmark workload, the drafter improved SGLang output throughput by 29%:

The early vLLM run measured:

• 20.14% token acceptance rate
• 1.81 accepted tokens per speculation step
• 51.27% acceptance at the first drafted position

There is substantial room to improve these results through serving configuration, draft-window tuning, higher acceptance rates, and workload-specific fine-tuning.

Why DFlash?

Large-model decoding can become memory-bandwidth-bound: each generated token requires reading the target model weights again. This can be especially relevant on VRAM-rich hardware where memory capacity is available but decoding bandwidth remains the bottleneck.

DFlash uses a smaller draft model to propose multiple candidate tokens, then verifies them with the target model. When proposals are accepted, the target model performs less sequential decoding work per output token.

Model details

• Architecture: DFlashDraftModel — DFlash
• Target / verifier model: Qwen/Qwen3.8-27B
• Parameters: ~1.7B
• Precision: BF16
• Vocabulary size: 248,320, shared with the target model
• Trained block size: 16
• Framework: SpecForge, from the SGLang team

Training

The model was trained with SpecForge on a single NVIDIA B300 GPU in two stages:

Training remained stable across both stages. GPU utilization stayed near 100%, with gradient norms generally in the ~0.2–0.6 range and no observed divergence.

This is a first release. The draft model and serving configurations are actively being benchmarked and tuned.

Usage

The commands below use 4 speculative tokens, which is the initial tested serving configuration. The model was trained with a block size of 16; benchmark and tune num_speculative_tokens for your workload rather than assuming one setting is universally best.

SGLang

python -m sglang.launch_server \
  --model-path Qwen/Qwen3.8-27B \
  --served-model-name qwen/qwen3.8 \
  --trust-remote-code \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path kstoyanov99/Qwen3.8-27B-Dflash \
  --speculative-num-draft-tokens 4 \
  --speculative-draft-attention-backend triton \
  --attention-backend flashinfer \
  --tp-size 1 \
  --context-length 240000 \
  --chunked-prefill-size 8196 \
  --mem-fraction-static 0.75 \
  --port 8000 \
  --host 0.0.0.0 \
  --mamba-scheduler-strategy extra_buffer \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

vLLM

vllm serve Qwen/Qwen3.8-27B \
  --served-model-name qwen/qwen3.8 \
  --max-model-len 200000 \
  --max-num-batched-tokens 8196 \
  --gpu-memory-utilization 0.88 \
  --port 8000 \
  --host 0.0.0.0 \
  --load-format fastsafetensors \
  --attention-backend flash_attn \
  --speculative-config '{"method":"dflash","num_speculative_tokens":4,"model":"kstoyanov99/Qwen3.8-27B-Dflash"}' \
  --enable-chunked-prefill \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --generation-config auto \
  --override-generation-config '{"temperature":0.7,"top_p":0.8,"top_k":20,"presence_penalty":0.0,"repetition_penalty":1.0}'

Notes

• These commands assume a recent SGLang or vLLM release with DFlash support.
• The target and draft model must use identical token IDs and vocabulary. Use the tokenizer from Qwen/Qwen3.8-27B.
• Adjust context length, memory fraction, batching, attention backend, and speculative-token count for your GPU and workload.
• flash_attn, flashinfer, and triton compatibility depends on your GPU architecture and installed inference-engine version.

Files

config.json          # DFlash draft-model configuration
model.safetensors    # Trained BF16 draft weights

Tokenizer files are intentionally not included. The target model tokenizer, Qwen/Qwen3.8-27B, should be used because the draft and target models must share a vocabulary and token IDs.

Limitations

• This is a draft model and cannot generate useful output by itself.
• It provides no benefit without the paired Qwen3.8-27B target model and DFlash-capable serving.
• Acceptance rate is workload-dependent. Higher acceptance generally produces higher speedups.
• The benchmarks above are early and limited; they are not a general performance guarantee.
• vLLM compatibility and serving configurations are still under active validation.

Roadmap

• Broader SGLang and vLLM benchmarks
• Acceptance-rate improvements
• Draft-window and block-size tuning
• Workload-specific fine-tuning experiments
• Reproducible training and serving recipes

Acknowledgements

Trained with SpecForge, built by the SGLang team, using the DFlash block-diffusion speculative-decoding method.