Qwen3.6-27B-NVFP4-GGUF

tngtech/Qwen3.6-27B-NVFP4-GGUF

<p align="center"> <img src="repack.png" alt="NVFP4 SafeTensors to GGUF repack illustration" width="600"> </p>

An NVFP4 GGUF repackage of nvidia/Qwen3.6-27B-NVFP4 that runs on llama.cpp's native Blackwell (sm_120) FP4 tensor-core path. nvidia/Qwen3.6-27B-NVFP4 is itself derived from Alibaba's Qwen3.6-27B.

Qwen3.6-27B is one of the strongest models that fits a 24 GB consumer GPU, delivering Sonnet-4.5-class coding intelligence at that size. Its published quants are almost all standard GGUF (Q4_K_M etc.), which do not use the FP4 hardware on RTX 5090 / Blackwell cards.

This repackage keeps the model's original NVFP4 4-bit weights bit-exact and hands them to llama.cpp's FP4 tensor cores, so you get the speed of hardware FP4 plus MTP speculative decoding, while staying inside 24 GB with real room left for KV cache. There is no measurable quality loss versus the BF16 model on our benchmarks.

---

Highlights

• ~50 % faster decode than Q4_K_M: 46.5 tok/s vs 31 tok/s on a 5090 laptop (24 GB), with MTP speculative decoding.
• Quality on par with full precision: matches BF16 on terminal-bench 2.0 and MMLU-Pro (within noise) on our tng-benchmark subset runs.
• Native FP4 on Blackwell: MLP and lm_head weights are repacked bit-exact from the source NVFP4 checkpoint (no re-rounding, no double quantization) and run on mma.sync FP4 tensor cores.
• Fits 24 GB with headroom: 19.41 GB on disk leaves ~4.5 GB for KV cache + draft + compute buffers.
• Big usable context: 160K context comfortably on 24 GB (native 262K), thanks to the hybrid Gated-DeltaNet + attention architecture (only 16 of 64 layers keep a growing KV cache).
• MTP built in: the model's Multi-Token-Prediction head (blk.64) is preserved, enabling draft-mtp self-speculation for the throughput numbers above.

---

Model description

How the weights are packed

The NVFP4 path avoids the double-quantization penalty of the usual "dequantize to F16 then re-quantize to Q4_K_M" route: the 4-bit weights are copied through untouched.

---

Files

This repository ships a single file:

• Qwen3.6-27B-NVFP4.gguf (19.41 GB): NVFP4 MLP and lm_head (bit-exact repack), Q8_0 attention/GDN projections, and Q8_0 token embedding and MTP head. 1252 tensors.

---

Usage (llama.cpp, deployment config for 24 GB)

Requires a llama.cpp build with sm_120 NVFP4 GEMM + CUDA Gated-DeltaNet (upstream ships both). The official CUDA server container works:

docker run --rm --gpus '"device=0"' \
    -v $HOME/models:/models -p 8080:8080 \
    ghcr.io/ggml-org/llama.cpp:server-cuda \
    -m /models/Qwen3.6-27B-NVFP4.gguf \
    -ngl 99 -c 160000 -fa on -np 1 \
    --cache-type-k q4_0 --cache-type-v q4_0 \
    --spec-draft-type-k q4_0 --spec-draft-type-v q4_0 \
    --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-ngl 99 \
    --temp 1 --top-p 0.95 --top-k 20 --presence-penalty 1.5 --min-p 0.0 \
    --port 8080 --host 0.0.0.0

Notes:

• --spec-type draft-mtp --spec-draft-n-max 3 is what unlocks the headline throughput: the MTP head drafts, the model verifies. n_max 3 is the sweet spot on this hardware.
• Sampling (--temp 1 --top-p 0.95 --top-k 20) matches the model's own generation_config.json. --presence-penalty 1.5 is aggressive anti-repetition; drop to 0.5 to 1.0 if you see token-avoidance in long structured output.
• -fa on (flash attention) is required for quantized KV and works with this model's head_dim 256.
• 160K < 262K native, so no RoPE scaling needed.

KV cache guidance (24 GB)

Only 16 layers keep a growing KV cache, so 160K fits, but it's tight alongside ~19 GB of weights. q4_0/q4_0 KV (~3 GB at 160K) is the safe default. K-cache precision matters more than V (K enters QK^T before softmax). If your prompts rarely exceed ~100K, --cache-type-k q8_0 --cache-type-v q4_0 with a lower -c buys better attention precision at the same VRAM.

---

Measurements

All measured on an RTX 5090 laptop (24 GB).

Size

Decode throughput (tok/s, with draft-mtp)

Q4_K_M peaks at n_max 1 (31 tok/s); this model tolerates deeper speculation and peaks at n_max 3. The FP4 weight path plus a well-behaved MTP head gives ~1.5x the throughput of the standard quant.

Quality (tng-benchmark suite)

Evaluated against full-precision bf16 on terminal-bench 2.0 and MMLU-Pro (TNG distilled v0.1 subsets) and the full IFBench. Run with --reasoning-preserve and f16 KV cache (best-case quality config). The Qwen3.6-27B-NVFP4 gguf repack row is the straight repackage (BF16 embedding + MTP), included to show that shrinking those to Q8_0 costs no quality.

Qwen3.6-27B-NVFP4.gguf matches BF16 on terminal-bench and is within noise on MMLU-Pro. Shrinking the token embedding and MTP head to Q8_0 did not cost quality (it scores at or above the straight repack on all three), consistent with the near-lossless Q8_0 embedding and the MTP head being draft-only.

Caveat: quality was measured with f16 KV cache; the q4_0/q4_0 deployment config above is expected to score somewhat lower. The distilled subsets and small sample counts also make single-point differences noisy.

---

How this was made

Converted CPU-only with a modified llama.cpp/convert_hf_to_gguf.py. We patched the converter to add the --embd-as-q8 and --mtp-as-q8 flags below (which force the token embedding and MTP layer to Q8_0), on top of the existing --fp8-as-q8 path:

export PYTHONPATH=$PWD/llama.cpp/gguf-py    # must be the repo-local gguf with GGML type NVFP4 (id 40)
python llama.cpp/convert_hf_to_gguf.py <SRC> \
    --outfile Qwen3.6-27B-NVFP4.gguf --fp8-as-q8 --embd-as-q8 --mtp-as-q8

• --fp8-as-q8: dequantize FP8 weights to Q8_0 (default would be larger BF16).
• --embd-as-q8: token embedding BF16 to Q8_0 (-1.16 GB, ~lossless).
• --mtp-as-q8: MTP layer BF16 to Q8_0 (-0.39 GB, draft-only).

---

Limitations and caveats

• Text only. The vision tower of the source VL model is not included. For multimodal use you would need a separate --mmproj.
• Requires FP4-capable hardware + build. Intended for Blackwell (sm_120, e.g. RTX 5090). On other GPUs the FP4 GEMM path won't engage and there is no benefit over a standard quant.
• Quality figures are the f16-KV best case. Real deployments using quantized KV cache will differ; benchmark subsets are distilled and noisy.
• general.name in the GGUF metadata is the HF snapshot hash (cosmetic).

---

License and attribution

Licensed under the Apache License 2.0, inherited from the upstream models. See the LICENSE and NOTICE files in this repository.

This is a format repackage (not a retrain) of nvidia/Qwen3.6-27B-NVFP4 (Apache 2.0), which is itself derived from Alibaba's Qwen3.6-27B (Apache 2.0). The only changes are quantization/format conversion to GGUF, documented under How this was made (--fp8-as-q8 --embd-as-q8 --mtp-as-q8).

Attribution: please credit the Qwen team (Alibaba) for the base model and NVIDIA for the NVFP4 (Model Optimizer) export, alongside this repackage.

Trademarks: "Qwen" and "NVIDIA" are trademarks of their respective owners. The Apache 2.0 license does not grant trademark rights; these names are used here only to identify the upstream models. This repackage is not affiliated with, sponsored by, or endorsed by NVIDIA or Alibaba.