Qwen3.8-Flash-Next-MTP-GGUF

Qwen3.8-Flash-Next MTP draft head (GGUF)

The NextN / MTP draft head from Qwen/Qwen3.8-Flash-Next, extracted and packaged as a standalone GGUF so it can be used as the draft model for speculative decoding with the full Qwen3.8-Flash-Next as the target.

This is not a standalone language model. It is a single prediction block (~2.6B parameters plus embeddings). It cannot generate text on its own and will produce nonsense if loaded as a normal model. It only does something useful when paired with the full Qwen3.8-Flash-Next as the target model.

Files

At temperature 0 both produce identical draft tokens on the prompts tested (181 drafted, 136 accepted, 75.1% acceptance for both), so the Q8_0 file costs nothing in draft quality and is the one to use.

Usage

The head is a draft model for llama-server:

llama-server \
  -m Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \
  -md qwen4exp-mtp-head-Q8_0.gguf \
  --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-n-min 1 \
  -ngl 99 -ngld 99

--spec-draft-n-max 2 is the measured optimum; deeper drafts accept less per position and lose more than the extra depth gains.

Requirements

These files were built with and are supported by mx-llama.cpp, a llama.cpp fork with qwen4exp MTP support, AMD gfx906 / MI50 tuning, tensor-parallel -sm tensor and a custom AllReduce. That is the build to use them with.

git clone https://github.com/mxxm-t/mx-llama.cpp

Support for this architecture is also being upstreamed - review is in progress at ggml-org/llama.cpp#27836. The tensor names here (blk.N.nextn.hc_head_*) match that convention, so these files are intended to keep working with upstream llama.cpp once it lands. Stock llama.cpp releases do not support qwen4exp MTP yet.

What acceptance to expect

Draft acceptance is strongly text-dependent, measured between roughly 46% and 90% across different prompts on the same model and settings. That variation is normal for this head, not a defect. Throughput follows acceptance closely, because a rejected draft is far more expensive than an accepted one.

Whether speculative decoding is a net win depends heavily on your hardware and split mode. On multi-GPU tensor-parallel setups the verify step can cost far more than the drafting saves. Measure on your own configuration before assuming a gain.

Provenance

Extracted from the original mtp.* tensors of Qwen3.8-Flash-Next and converted with convert_hf_to_gguf.py --mtp. No weights were retrained, fine-tuned or otherwise altered - only re-quantized (Q8_0) or format-converted (BF16).

License

Released under the Qwen Community License 1.0, inherited from the base model. A copy of the licence is included in this repository as required by its terms. Note clause 2: a separate licence from Qwen is required for Model as a Service or AI Work Assistant commercial use.