Qwen3.8-27B-DFlash2-ROCmFP4-FAST-GGUF

Qwen3.8-27B DFlash2 draft model, ROCmFP4-FAST GGUF

A 4.25 bpw ROCmFP4 requantisation of z-lab/Qwen3.8-27B-DFlash2, for use as a speculative decoding sidecar with a Qwen3.8-27B target.

Measured on an AMD Strix Halo (Radeon 8060S), paired with julianmb/Qwen-3.8-27B-ROCmFP4-FAST-GGUF: 65.6 t/s on structured output, 4.7x bare decode.

Two things to know before you download

1. This will not load in stock llama.cpp. Q4_0_ROCMFP4_FAST is GGUF file type 103, which exists only in the fork linked below. Upstream llama.cpp fails with failed to load model before it reaches the GPU. Get a prebuilt binary here:

• https://github.com/LaurentZuijdwijk/llama.cpp/releases

2. This is a draft model, not a standalone one. It has no full graph of its own. Loading it by itself fails with failed to create context, which is expected. It must be passed with -md alongside a target model.

Usage

llama-server \
  -m  Qwen3.8-27B-ROCmFP4-FAST.gguf \
  -md Qwen3.8-27B-DFlash2-Q4_0_ROCMFP4_FAST.gguf \
  --spec-type draft-dflash --spec-draft-adaptive \
  --spec-draft-n-min 3 --spec-draft-n-max 7 --spec-draft-ngl 99 \
  -ngl 999 -fa on -b 2048 -ub 512 -c 32768

--spec-draft-adaptive sizes the draft from measured acceptance instead of a fixed n. It is what makes n-max 7 safe here: at a fixed n=7 acceptance collapses to 18 % and throughput drops to 20.2 t/s, while adaptive holds 96 % acceptance and reaches 65.6.

Speculative decoding raises throughput, not quality: the target model verifies every token, so output matches what the target would have produced on its own.

Requirements

• The fork above, or any build with the ROCmFPx quant types
• A Vulkan 1.3 driver. Measured on Mesa RADV 26.0.8 on gfx1151.
• The FP4 path is tuned for AMD Strix Halo. It should run anywhere the fork builds, but the numbers above are specific to this hardware.

A note on requantising

Do not requantise a DFlash2 sidecar to Q8_0_ROCMFPX expecting parity with Q8_0. At identical bits per weight, ours scored 53.5 % acceptance against z-lab's 60.2 %. The cause is the block scale, not the codes: Q8_0 stores an fp16 scale, ROCmFPx stores a UE4M3 byte. At 8 bits per weight the coarse scale is what costs you. FP4 is a different trade and is the one measured here.

Credits

• z-lab/Qwen3.8-27B-DFlash2, the source model (Apache-2.0)
• ciru-ai/ROCmFPX for the FPx quant formats
• DFlash2 support in llama.cpp, PR #27342 by Jian Chen
• ggml-org/llama.cpp

Quantized and published by Agention.

Licensed Apache-2.0, inherited from the base model.