Qwen3.5-9B-DFlash-GGUF

Qwen3.5-9B-DFlash — GGUF Quantized Draft Model

DFlash Speculative Decoding optimized for Speculative Decoding acceleration with target models.

This repository contains the GGUF quantized versions of z-lab/Qwen3.5-9B-DFlash, a Dense architecture draft model designed exclusively for Speculative Decoding inference acceleration.

The DFlash model serves as a lightweight draft predictor that rapidly generates candidate tokens, which are then verified by a larger target model (e.g., Qwen3.5-9B). This two-stage approach dramatically improves token generation throughput without sacrificing output quality.

---

Compatibility

Requires a recent version of llama.cpp with native DFlash support.

Tested with:

• llama.cpp b9831 or newer

Model Summary

---

Quantization Summary

Quick Reference

• 🏆 Recommended (Sweet Spot): Q5_K_M — Best balance of accuracy and speed for 16GB VRAM setups
• 💾 Smallest: Q4_K_M — Fits comfortably on any GPU; marginal quality loss
• 🔬 Highest Fidelity: BF16 / Q8_0 — Near-original precision; useful for benchmarking

---

Quantization Workflow

All quantizations are performed using llama.cpp’s ./llama-quantize tool. The workflow starts from the BF16 master copy and produces four quantized variants:

Step 1: BF16 Master (Reference)

# Ensure you have a clean BF16 source file
cp qwen3.5-9b-dflash-bf16.gguf ./source-bf16.gguf

Step 2: Generate Quantized Versions

./llama-quantize ./source-bf16.gguf \
  ./output/qwen3.5-9b-dflash-Q4_K_M.gguf Q4_K_M

./llama-quantize ./source-bf16.gguf \
  ./output/qwen3.5-9b-dflash-q5_k_m.gguf Q5_K_M

./llama-quantize ./source-bf16.gguf \
  ./output/qwen3.5-9b-dflash-Q6_K.gguf Q6_K

./llama-quantize ./source-bf16.gguf \
  ./output/qwen3.5-9b-dflash-Q8_0.gguf Q8_0

Platform Notes

★ On Windows PowerShell, use semicolons (;) or the backtick escape character to chain commands. On macOS/Linux, use && or backslash-newline.

---

How to Use

Speculative Decoding with llama-server

llama-server \
  --model Qwen3.5-9B.gguf \
  --spec-draft-model qwen3.5-9b-dflash-q5_k_m.gguf \
  --spec-type draft-dflash \
  --spec-draft-n-max 3 \
  --host 127.0.0.1 \
  --port 8080

Via llama-cli (Interactive)

./llama-cli \
  -m Qwen3.5-9B.gguf \
  --spec-draft-model qwen3.5-9b-dflash-q5_k_m.gguf \
  --spec-type draft-dflash \
  --spec-draft-n-max 3 \
  -p "Write a concise summary of..." \
  -n 512

VRAM Optimization Tips

---

Acknowledgments

• z-lab — Original DFlash model architecture and implementation
• Qwen Team — Qwen3.5-9B base model
• llama.cpp / ggml-org — GGUF format, DFlash inference engine, and quantization toolchain

---

License

This repository contains converted GGUF quantizations of the original DFlash draft model. All original licenses, usage restrictions, and intellectual property remain with the upstream authors. Refer to the original HuggingFace model card for complete licensing information.

---

Quantized for speculative decoding acceleration. Always pair with a compatible target model.