Qwen3.8-27B-NVFP4-MTP-GGUF

Qwen3.8-27B NVFP4-MTP GGUF

NVFP4 quantized Qwen3.8-27B with MTP (multi-token prediction) draft layers, packaged as a single GGUF for llama.cpp speculative decoding on Blackwell GPUs (RTX 5090 / 5080).

What this is

• Quantization: NVFP4 (weights + KV cache), converted to GGUF via llama.cpp convert_hf_to_gguf.py --outtype auto
• MTP draft layers included — enables --spec-type draft-mtp --spec-draft-n-max 2 in llama.cpp for speculative decoding
• Multimodal: pairs with the mmproj-F16.gguf vision projector (from unsloth/Qwen3.8-27B-GGUF)
• Single file: qwen3.8-27b-text-nvfp4-mtp.gguf (~19 GB)

Attribution

• Source weights: sakamakismile/Qwen3.8-27B-MTP-NVFP4 (Apache-2.0) — this repo is a mechanical GGUF conversion of that repo's safetensors; all credit for the NVFP4 quantization goes to sakamakismile
• Base model: Qwen/Qwen3.8-27B

Conversion methodology

Converted on an RTX 5090 (CachyOS, user-space, no Docker). The conversion script lives in the felippeburk/rtx-5090-cachyos-testing repo:

# Builds llama.cpp with NVFP4/MTP support, downloads source safetensors,
# runs the conversion, and verifies the result. Everything installs to ~/.local.
git clone https://gitlab.com/felippeburk/rtx-5090-cachyos-testing
cd rtx-5090-cachyos-testing
bash llama/setup-mtp.sh --model qwen38

The conversion step is: llama.cpp/convert_hf_to_gguf.py <source-safetensors-dir> --outfile qwen3.8-27b-text-nvfp4-mtp.gguf --outtype auto, run with torch + transformers in an isolated venv. Reproducible from the source safetensors at any time.

Recommended server args (llama.cpp, 196K context — daily driver)

-m qwen3.8-27b-text-nvfp4-mtp.gguf \
--mmproj mmproj-F16.gguf \
-fitt 8192 -c 196608 -n 65536 -fa on -ngl 99 -np 1 -t 16 -tb 16 \
-ctk q8_0 -ctv q8_0 -ctkd q4_1 -ctvd q4_1 -ctxcp 16 -cram 6144 \
--cache-idle-slots --no-warmup \
--spec-type draft-mtp --spec-draft-n-max 2 \
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
--presence-penalty 0.0 --repeat-penalty 1.0 \
--jinja --chat-template-file config/chat_template.jinja \
--reasoning on --reasoning-format deepseek --reasoning-budget 4096 \
--reasoning-budget-message "I have reached my reasoning budget. I will now provide my final answer or take the most appropriate action based on my analysis so far." \
--chat-template-kwargs '{"reasoning_effort":"xhigh"}'

Important usage notes

• Thinking mode requires a generous output budget. This model uses the Qwen3.8 thinking format, so reasoning shares the max_tokens budget with the answer. Set the client's max_tokens to at least 65536 (-n 65536 above) and give the server a --reasoning-budget (here 4096) so the thinking phase can't eat the entire output. With a too-small max_tokens, long prompts can produce reasoning but an empty final answer.
• Pin a reasoning-effort budget message. --reasoning-budget-message tells the model to wrap up and answer once the budget is hit; without it, long tool-calling sessions can stall in thinking.
• Use thinking-mode sampling. Temperature 1.0, presence_penalty 0.0 (the 0.7 / 1.5 values in some older configs are for instruct mode and hurt thinking-mode output).
• reasoning_effort only accepts xhigh / medium / low — there is no high level. The patched chat template defaults to medium; the recommended config forces xhigh via --chat-template-kwargs.
• Vision is available via --mmproj mmproj-F16.gguf (downloaded from unsloth/Qwen3.8-27B-GGUF).

Full config (131K tier, plus the 262K experimental tier) lives in config/models.yaml of that GitLab repo.

License

Apache-2.0 (inherited from source weights and base model)