Downloads · 30 days
24
1% of all-time downloads
utautako/Qwen3.6-27B-NVIDIA-NVFP4-MTP-Q8attn-GGUF
Qwen3.6-27B-NVIDIA-NVFP4-MTP-Q8attn-GGUF is a image-text-to-text model from utautako. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for llama.cpp. The card lists the license as apache-2.0.
GGUF conversion of nvidia/Qwen3.6-27B-NVFP4, preserving NVIDIA's NVFP4 MLP tensors, with MTP speculative decoding and a BF16 vision projector.
Downloads · 30 days
24
1% of all-time downloads
All-time downloads
3.1K
Public
Repo size
20.1 GB
Likes
2
Public
Click a slice to open those files.
.gguf20.1 GB · 100%
From the Hugging Face model README
GGUF conversion of nvidia/Qwen3.6-27B-NVFP4, preserving NVIDIA's NVFP4 MLP tensors, with MTP speculative decoding and a BF16 vision projector.
This is the accuracy-oriented variant. Compared to the smaller
Q4-attention build, this one keeps
the attention and linear-attention (DeltaNet) projections at Q8_0 — matching the original FP8
precision of the NVIDIA checkpoint — instead of dropping them to Q4_K. That adds about 3 GB to
the file size, with the aim of reducing accuracy loss at long context, where 4-bit attention /
DeltaNet projections hurt most. Benchmarked on an RTX 5090 with llama-benchy.
q/k/v/o) and linear-attention / DeltaNet projections (attn_qkv, attn_gate, ssm_out, ssm_alpha, ssm_beta) are stored as Q8_0 (≈ the original FP8), for better accuracy than the Q4-attention build.draft-mtp speculative decoding.mmproj file as the Q4-attention build for image input.llama-benchy using MTP depth d=3.| Build | Attention / DeltaNet precision | Size | Pick it for |
|---|---|---|---|
| Q4-attention (sibling) | Q4_K | 14.66 GiB | Smallest footprint |
| Q8attn (this repo) | Q8_0 | 17.81 GiB | Better accuracy, esp. long context |
Both builds share the same 193 NVFP4 MLP tensors, MTP layer, and BF16 vision projector. Generation
speed with MTP d=3 is comparable (~110–130 tok/s on RTX 5090), so the choice is accuracy vs. size,
not throughput.
| Component | Source |
|---|---|
| Base model | Qwen/Qwen3.6-27B |
| NVFP4 source checkpoint | nvidia/Qwen3.6-27B-NVFP4 |
| Runtime target | llama.cpp |
| File | Size | Description |
|---|---|---|
Qwen3.6-27B-NVIDIA-NVFP4-MTP-Q8attn.gguf | 17.81 GiB / 19,128,366,464 bytes | Main GGUF. NVFP4 MLP preserved; attention/DeltaNet at Q8_0; MTP layer included. |
mmproj-Qwen3.6-27B-NVIDIA-NVFP4-BF16.gguf | ~888 MiB / 931,146,304 bytes | BF16 vision projector for image input. |
# Text-only
llama-server \
-m Qwen3.6-27B-NVIDIA-NVFP4-MTP-Q8attn.gguf \
--spec-type draft-mtp \
--spec-draft-n-max 3 \
-c 196608 \
-ngl 999
# With vision
llama-server \
-m Qwen3.6-27B-NVIDIA-NVFP4-MTP-Q8attn.gguf \
--mmproj mmproj-Qwen3.6-27B-NVIDIA-NVFP4-BF16.gguf \
--spec-type draft-mtp \
--spec-draft-n-max 3 \
-c 147456 \
-ngl 999
The context lengths above are what this repo's author runs on a 32 GB RTX 5090:
-c 196608 for text-only and -c 147456 when the vision projector is loaded (the mmproj plus
its image tokens take extra VRAM, so a smaller context leaves headroom). Adjust to fit your own VRAM.
Measured on RTX 5090, llama-benchy, MTP depth d=3, ctx 32768, generation-latency mode.
(Absolute tok/s is not 1:1 comparable to the sibling card's table: this run used a different
llama.cpp build and a gpt2 tokenizer fallback. Use it to compare thinking on/off within this table.)
| Mode | Prompt / gen length | Generation (tg) tok/s | Peak gen tok/s |
|---|---|---|---|
| thinking ON | pp512 / tg512 | 125.28 ± 2.67 | ~135 |
| thinking ON | pp4096 / tg512 | 127.05 ± 0.96 | ~135 |
| thinking OFF | pp512 / tg512 | 115.35 ± 10.30 | — |
| thinking OFF | pp4096 / tg512 | 110.04 ± 9.12 | — |
MTP d=3 roughly doubles generation throughput over the non-speculative baseline on this hardware.
| Field | Value |
|---|---|
| GGUF architecture | qwen35 |
| License metadata | apache-2.0 |
| Text layers | 64 |
| MTP layers | 1 |
qwen35.block_count | 65 |
qwen35.nextn_predict_layers | 1 |
| Bits per weight | ~5.60 BPW |
| Vision projector architecture | clip / qwen3vl_merger |
| Vision projector precision | BF16/F32 |
| Tensor type | Count | Role |
|---|---|---|
| NVFP4 | 193 | MLP (gate/up/down) — preserved from the NVIDIA NVFP4 checkpoint |
| Q8_0 | 308 | attention (q/k/v/o) + linear-attn / DeltaNet projections |
| Q6_K | 3 | MTP-layer FFN |
| Q4_K | 2 | remaining small tensors |
| F32 | 746 | norms / embeddings / scales |
| Total | 1252 |
The conversion keeps the original NVFP4 MLP tensors instead of requantizing all weights to K-quants,
and, unlike the Q4-attention build, keeps attention / DeltaNet projections at Q8_0 rather than Q4_K.
The upstream models are licensed under Apache-2.0:
This repository redistributes a GGUF format conversion of the NVIDIA NVFP4 checkpoint under the same Apache-2.0 license. Please also follow the terms and usage guidance of the upstream model cards.
Thanks to the Qwen team for the base model, NVIDIA for the NVFP4 ModelOpt checkpoint, and the
llama.cpp project for GGUF, NVFP4, MTP, and multimodal runtime support.