Qwen3.6-27B-NVFP4-GGUF

Qwen3.6-27B-NVFP4-GGUF

GGUF conversion of nvidia/Qwen3.6-27B-NVFP4 for use with llama.cpp.

This is an NVFP4-quantized standalone language model.

The converted GGUF preserves the NVFP4 tensors and the model's native MTP tensors. It is not a separate draft model and does not require an external speculator.

Model Details

• Source model: nvidia/Qwen3.6-27B-NVFP4
• Base model: Qwen/Qwen3.6-27B
• Format: GGUF
• Quantization: NVIDIA NVFP4 / ModelOpt
• Architecture: Qwen3.6 27B dense model
• Purpose: Local inference and native MTP speculative decoding with llama.cpp

NVIDIA quantized the weights and activations of linear operators inside the transformer blocks to NVFP4. Other tensors may remain in higher-precision formats.

This repository contains the complete target model. It is not an MTP, EAGLE3, or DFlash draft-only checkpoint.

Compatibility

A recent version of llama.cpp with Qwen3.6, NVFP4, and native MTP support is required.

Tested with:

• Windows
• NVIDIA GeForce RTX 5070 Ti 16 GB
• NVIDIA GeForce RTX 5060 Ti 16 GB
• llama.cpp CUDA backend
• Native Qwen3.6 MTP speculative decoding

Older llama.cpp builds may fail to recognize the nvfp4 tensor type or may not correctly load the associated scale tensors.

Performance may vary with llama.cpp build, GPU split, context size, KV-cache format, prompt, sampling settings, and other runtime options.

Usage

llama-server

llama-server \
  -m Qwen3.6-27B-NVFP4.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --flash-attn on \
  -ngl 99

llama-cli

llama-cli \
  -m Qwen3.6-27B-NVFP4.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --flash-attn on \
  -ngl 99

Multi-GPU example

llama-server \
  -m Qwen3.6-27B-NVFP4.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 3 \
  --split-mode layer \
  --tensor-split 1,1 \
  --flash-attn on \
  -ngl 99

The best tensor split depends on available VRAM, GPU speed, PCIe topology, context length, and KV-cache placement. An even split is only a starting point.

Suggested Settings

Based on the mixed-task benchmark below:

• n_max = 3 gave the best aggregate wall-clock result of the tested settings.
• n_max = 2 gave the highest aggregate acceptance rate.
• n_max = 3 was substantially faster for code completion, JSON, repeated patterns, summarization, mathematical reasoning, and long reasoning.
• n_max = 2 performed better for translation, creative writing, and conceptual explanations.
• High-entropy natural-language tasks lose acceptance more quickly as the draft length increases.

A reasonable general-purpose starting point is:

--spec-draft-n-max 3

For translation, role-play, creative writing, conversational output, or open-ended explanations, start with:

--spec-draft-n-max 2

For code, JSON, tool output, repeated patterns, and other predictable generation, n_max = 3 is the better tested setting.

Conversion

Converted from the original Hugging Face ModelOpt checkpoint using a recent convert_hf_to_gguf.py from llama.cpp:

python convert_hf_to_gguf.py \
  /path/to/Qwen3.6-27B-NVFP4 \
  --outfile Qwen3.6-27B-NVFP4.gguf \
  --outtype auto \
  --verbose

Do not pass a conventional GGUF quantization type such as q4_k_m when the goal is to preserve the original NVFP4 tensors. The conversion repackages the ModelOpt checkpoint into GGUF rather than requantizing it into a K-quant.

No neural-network weights were retrained. The original NVFP4 weights and their associated scale tensors were converted to GGUF-compatible representations.

Verifying the GGUF

The converted file can be checked with llama-gguf:

.\llama-gguf.exe E:\HF_MODELS\Qwen3.6-27B-NVFP4.gguf r

A successful conversion should show NVFP4 tensors and their associated scale tensors, for example:

tensor: name = blk.0.ffn_down.weight,       type = nvfp4
tensor: name = blk.0.ffn_down.scale,        type = f32
tensor: name = blk.0.ffn_down.input_scale,  type = f32

This confirms that the checkpoint was not converted into a conventional Q4 type and is not an empty metadata-only GGUF.

Benchmark

Benchmarked using mtp-bench with native Qwen3.6 MTP speculative decoding.

Aggregate Results

Effective batch throughput is calculated from total predicted tokens divided by total wall time. Per-request tok/s values may be higher because aggregate wall time can also include prompt processing and benchmark overhead.

These results do not include a non-speculative baseline and therefore should not be interpreted as a direct speedup ratio over standard decoding.

Detailed Results

n_max = 2

code_python        pred= 192 draft= 132 acc= 124 rate=0.939 tok/s=44.0
code_cpp           pred=  54 draft=  36 acc=  36 rate=1.000 tok/s=46.1
explain_concept    pred= 192 draft= 173 acc= 104 rate=0.601 tok/s=34.2
summarize          pred=  46 draft=  34 acc=  29 rate=0.853 tok/s=41.3
qa_factual         pred= 170 draft= 132 acc= 103 rate=0.780 tok/s=39.8
translation        pred=  17 draft=  12 acc=  10 rate=0.833 tok/s=42.4
creative_short     pred=  35 draft=  34 acc=  18 rate=0.529 tok/s=31.6
stepwise_math      pred= 192 draft= 138 acc= 121 rate=0.877 tok/s=42.4
json_output        pred= 192 draft= 128 acc= 127 rate=0.992 tok/s=46.2
long_reasoning     pred= 192 draft= 138 acc= 122 rate=0.884 tok/s=42.6
repeat_pattern     pred= 192 draft= 127 acc= 127 rate=1.000 tok/s=46.0
code_completion    pred= 191 draft= 130 acc= 126 rate=0.969 tok/s=45.1
long_code_review   pred= 192 draft= 166 acc= 108 rate=0.651 tok/s=35.6

Aggregate:
  requests:            13
  predicted tokens:    1857
  draft tokens:        1380
  accepted tokens:     1155
  acceptance rate:     0.8370
  total wall time:     48.97 s

n_max = 3

code_python        pred= 192 draft= 151 acc= 140 rate=0.927 tok/s=50.1
code_cpp           pred=  54 draft=  45 acc=  40 rate=0.889 tok/s=47.8
explain_concept    pred= 192 draft= 232 acc= 113 rate=0.487 tok/s=33.2
summarize          pred=  46 draft=  39 acc=  32 rate=0.821 tok/s=47.3
qa_factual         pred= 170 draft= 162 acc= 115 rate=0.710 tok/s=42.7
translation        pred=  17 draft=  18 acc=  12 rate=0.667 tok/s=36.7
creative_short     pred=  35 draft=  48 acc=  21 rate=0.438 tok/s=28.6
stepwise_math      pred= 192 draft= 165 acc= 135 rate=0.818 tok/s=46.4
json_output        pred= 192 draft= 144 acc= 142 rate=0.986 tok/s=52.6
long_reasoning     pred= 192 draft= 167 acc= 135 rate=0.808 tok/s=46.4
repeat_pattern     pred= 192 draft= 144 acc= 142 rate=0.986 tok/s=53.5
code_completion    pred= 191 draft= 147 acc= 141 rate=0.959 tok/s=53.0
long_code_review   pred= 192 draft= 218 acc= 118 rate=0.541 tok/s=35.5

Aggregate:
  requests:            13
  predicted tokens:    1857
  draft tokens:        1680
  accepted tokens:     1286
  acceptance rate:     0.7655
  total wall time:     46.00 s

Observations

• n_max = 3 reduced total wall time by approximately 6.1% compared with n_max = 2.
• n_max = 3 increased effective aggregate throughput from approximately 37.92 tok/s to 40.37 tok/s.
• n_max = 2 retained a very high aggregate acceptance rate of 83.7%.
• Code completion improved from 45.1 tok/s to 53.0 tok/s with n_max = 3.
• JSON output improved from 46.2 tok/s to 52.6 tok/s.
• Repeated-pattern generation improved from 46.0 tok/s to 53.5 tok/s.
• Translation declined from 42.4 tok/s to 36.7 tok/s.
• Creative writing declined from 31.6 tok/s to 28.6 tok/s.
• Conceptual explanation and long code review received little or no benefit from increasing the draft length.

The results show a clear task-dependent trade-off: predictable outputs benefit from a longer native MTP draft, while high-entropy natural-language generation generally favors a shorter draft.

Notes

• This GGUF preserves NVIDIA's NVFP4 tensor type; it is not equivalent to Q4_K_M, Q4_K_S, or an Unsloth dynamic quant.
• The source checkpoint was officially prepared for ModelOpt and vLLM. llama.cpp support is a separate community implementation and may behave differently from NVIDIA's reference runtime.
• Model size, VRAM requirements, and speed should not be estimated as though every parameter were stored as a plain 4-bit scalar. NVFP4 uses block scales and leaves some tensors at higher precision.
• Native MTP accelerates token generation but does not improve prompt-prefill speed in the same way.
• Very short benchmark outputs, especially translation and C++ in this run, are more sensitive to run-to-run variance.
• Results are specific to the tested hardware, llama.cpp build, prompts, runtime options, and context configuration.

Credits

• Qwen Team / Alibaba Cloud — Qwen3.6-27B
• NVIDIA — ModelOpt and the original NVFP4 checkpoint
• ggml-org — llama.cpp, GGUF, NVFP4 inference support, and native MTP support

License

The source model is distributed under the Apache License 2.0.

Users should review the upstream nvidia/Qwen3.6-27B-NVFP4 and Qwen/Qwen3.6-27B model cards before redistribution or commercial use.