Downloads · 30 days
157
3% of all-time downloads
williamliao/Qwen3.6-35B-A3B-NVFP4-GGUF
Qwen3.6-35B-A3B-NVFP4-GGUF is a text generation model from williamliao. Use it when you need the model to write or continue text. It is set up for llama.cpp. The card lists the license as apache-2.0.
GGUF conversion of nvidia/Qwen3.6-35B-A3B-NVFP4 for use with llama.cpp.
Downloads · 30 days
157
3% of all-time downloads
All-time downloads
5.2K
Public
Repo size
46.5 GB
Likes
7
Public
Click a slice to open those files.
.gguf22.7 GB · 100%
From the Hugging Face model README
GGUF conversion of nvidia/Qwen3.6-35B-A3B-NVFP4 for use with llama.cpp.
This is an NVFP4-quantized standalone Mixture-of-Experts language model.
The converted GGUF preserves the NVFP4 tensors and the model's native MTP tensors. It is not a separate draft model and does not require an external speculator.
nvidia/Qwen3.6-35B-A3B-NVFP4Qwen/Qwen3.6-35B-A3BNVIDIA quantized the weights and activations of linear operators inside the MoE transformer blocks to NVFP4. Other tensors may remain in higher-precision formats.
This repository contains the complete target model. It is not an MTP, EAGLE3, or DFlash draft-only checkpoint.
A recent version of llama.cpp with Qwen3.6 MoE, NVFP4, and native MTP support is required.
Tested with:
Older llama.cpp builds may fail to recognize the nvfp4 tensor type, may not correctly load associated scale tensors, or may lack compatible Qwen3.6 MoE support.
Performance may vary with llama.cpp build, GPU split, context size, KV-cache format, prompt, sampling settings, and other runtime options.
llama-server \
-m Qwen3.6-35B-A3B-NVFP4.gguf \
--spec-type draft-mtp \
--spec-draft-n-max 3 \
--flash-attn on \
-ngl 99
llama-cli \
-m Qwen3.6-35B-A3B-NVFP4.gguf \
--spec-type draft-mtp \
--spec-draft-n-max 3 \
--flash-attn on \
-ngl 99
llama-server \
-m Qwen3.6-35B-A3B-NVFP4.gguf \
--spec-type draft-mtp \
--spec-draft-n-max 3 \
--split-mode layer \
--tensor-split 1,1 \
--flash-attn on \
-ngl 99
The best tensor split depends on available VRAM, GPU speed, PCIe topology, context length, and KV-cache placement. An even split is only a starting point.
Based on the mixed-task benchmark below:
n_max = 3 gave the best aggregate result of the tested settings.n_max = 2 gave the highest aggregate acceptance rate and was clearly better for translation and other high-entropy natural-language output.n_max = 4 produced the highest per-request throughput for highly predictable output such as repeated patterns, JSON, and code completion, but did not improve aggregate throughput over n_max = 3.n_max = 3 sharply reduced acceptance on creative writing, conceptual explanation, translation, and long code review.A reasonable general-purpose starting point is:
--spec-draft-n-max 3
For translation, role-play, creative writing, conversational output, or open-ended explanations, start with:
--spec-draft-n-max 2
For JSON, fixed templates, repeated patterns, and deterministic code completion, try:
--spec-draft-n-max 4
Converted from the original Hugging Face ModelOpt checkpoint using a recent convert_hf_to_gguf.py from llama.cpp:
python convert_hf_to_gguf.py \
/path/to/Qwen3.6-35B-A3B-NVFP4 \
--outfile Qwen3.6-35B-A3B-NVFP4.gguf \
--outtype auto \
--verbose
Do not pass a conventional GGUF quantization type such as q4_k_m when the goal is to preserve the original NVFP4 tensors. The conversion repackages the ModelOpt checkpoint into GGUF rather than requantizing it into a K-quant.
No neural-network weights were retrained. The original NVFP4 weights and their associated scale tensors were converted to GGUF-compatible representations.
The converted GGUF may be larger than the original Hugging Face ModelOpt checkpoint.
The conversion does not copy the original safetensors byte-for-byte. NVFP4 weights are repacked into the GGML layout, and the GGUF stores the scale and input-scale tensors required by llama.cpp.
A larger GGUF file does not mean that the model was converted to FP16 or to a conventional Q4 format. The tensor dump should still show the main linear weights as:
type = nvfp4
Windows reports file sizes using binary units even though File Explorer labels them as GB. For example, a file shown as 28.2 GB by Hugging Face may appear as approximately 26.2 GB in Windows.
File size should not be treated as the exact amount of VRAM required at runtime. Additional memory is needed for the KV cache, CUDA context, compute buffers, graph workspace, and native MTP execution.
The converted file can be checked with llama-gguf:
.\llama-gguf.exe E:\HF_MODELS\Qwen3.6-35B-A3B-NVFP4.gguf r
A successful conversion should show NVFP4 tensors and their associated scale tensors, for example:
tensor: name = blk.0.ffn_down.weight, type = nvfp4
tensor: name = blk.0.ffn_down.scale, type = f32
tensor: name = blk.0.ffn_down.input_scale, type = f32
This confirms that the checkpoint was not converted into a conventional Q4 type and is not an empty metadata-only GGUF.
Benchmarked using mtp-bench with native Qwen3.6 MTP speculative decoding.
n_max | Draft acceptance | Predicted tokens | Draft tokens | Accepted tokens | Wall time | Effective batch throughput |
|---|---|---|---|---|---|---|
| 2 | 82.2% | 1,824 | 1,370 | 1,126 | 15.83 s | 115.22 tok/s |
| 3 | 72.8% | 1,807 | 1,690 | 1,230 | 14.81 s | 122.01 tok/s |
| 4 | 64.2% | 1,804 | 2,015 | 1,294 | 14.91 s | 121.00 tok/s |
Effective batch throughput is calculated from total predicted tokens divided by total wall time. Per-request tok/s values may be higher because aggregate wall time can also include prompt processing and benchmark overhead.
The predicted-token totals differ slightly between runs because some test outputs terminated at different lengths. Aggregate throughput is therefore more useful than wall time alone when comparing these runs.
These results do not include a non-speculative baseline and therefore should not be interpreted as a direct speedup ratio over standard decoding.
n_max = 2code_python pred= 192 draft= 132 acc= 125 rate=0.947 tok/s=149.5
code_cpp pred= 54 draft= 36 acc= 36 rate=1.000 tok/s=151.5
explain_concept pred= 192 draft= 172 acc= 104 rate=0.605 tok/s=117.4
summarize pred= 50 draft= 38 acc= 31 rate=0.816 tok/s=135.7
qa_factual pred= 192 draft= 142 acc= 120 rate=0.845 tok/s=143.9
translation pred= 17 draft= 12 acc= 10 rate=0.833 tok/s=134.3
creative_short pred= 39 draft= 38 acc= 20 rate=0.526 tok/s=107.6
stepwise_math pred= 192 draft= 135 acc= 123 rate=0.911 tok/s=147.5
json_output pred= 192 draft= 132 acc= 124 rate=0.939 tok/s=146.5
long_reasoning pred= 192 draft= 145 acc= 118 rate=0.814 tok/s=138.7
repeat_pattern pred= 192 draft= 127 acc= 127 rate=1.000 tok/s=153.9
code_completion pred= 128 draft= 86 acc= 85 rate=0.988 tok/s=153.8
long_code_review pred= 192 draft= 175 acc= 103 rate=0.589 tok/s=113.4
Aggregate:
requests: 13
predicted tokens: 1824
draft tokens: 1370
accepted tokens: 1126
acceptance rate: 0.8219
total wall time: 15.83 s
n_max = 3code_python pred= 192 draft= 150 acc= 141 rate=0.940 tok/s=167.6
code_cpp pred= 54 draft= 45 acc= 39 rate=0.867 tok/s=155.6
explain_concept pred= 192 draft= 232 acc= 112 rate=0.483 tok/s=112.3
summarize pred= 50 draft= 48 acc= 33 rate=0.688 tok/s=134.5
qa_factual pred= 192 draft= 174 acc= 132 rate=0.759 tok/s=149.8
translation pred= 17 draft= 21 acc= 11 rate=0.524 tok/s=103.8
creative_short pred= 39 draft= 48 acc= 22 rate=0.458 tok/s=107.2
stepwise_math pred= 192 draft= 166 acc= 135 rate=0.813 tok/s=153.3
json_output pred= 192 draft= 156 acc= 139 rate=0.891 tok/s=162.3
long_reasoning pred= 192 draft= 181 acc= 130 rate=0.718 tok/s=141.6
repeat_pattern pred= 192 draft= 144 acc= 143 rate=0.993 tok/s=176.8
code_completion pred= 111 draft= 87 acc= 83 rate=0.954 tok/s=168.8
long_code_review pred= 192 draft= 238 acc= 110 rate=0.462 tok/s=109.3
Aggregate:
requests: 13
predicted tokens: 1807
draft tokens: 1690
accepted tokens: 1230
acceptance rate: 0.7278
total wall time: 14.81 s
n_max = 4code_python pred= 192 draft= 168 acc= 148 rate=0.881 tok/s=170.2
code_cpp pred= 54 draft= 48 acc= 44 rate=0.917 tok/s=167.8
explain_concept pred= 192 draft= 293 acc= 117 rate=0.399 tok/s=110.0
summarize pred= 50 draft= 52 acc= 37 rate=0.712 tok/s=149.3
qa_factual pred= 192 draft= 212 acc= 137 rate=0.646 tok/s=145.8
translation pred= 17 draft= 28 acc= 12 rate=0.429 tok/s=96.1
creative_short pred= 36 draft= 72 acc= 20 rate=0.278 tok/s=84.8
stepwise_math pred= 192 draft= 187 acc= 143 rate=0.765 tok/s=159.9
json_output pred= 192 draft= 174 acc= 147 rate=0.845 tok/s=171.5
long_reasoning pred= 192 draft= 215 acc= 137 rate=0.637 tok/s=142.8
repeat_pattern pred= 192 draft= 153 acc= 152 rate=0.994 tok/s=191.2
code_completion pred= 111 draft= 100 acc= 88 rate=0.880 tok/s=177.6
long_code_review pred= 192 draft= 313 acc= 112 rate=0.358 tok/s=101.4
Aggregate:
requests: 13
predicted tokens: 1804
draft tokens: 2015
accepted tokens: 1294
acceptance rate: 0.6422
total wall time: 14.91 s
n_max = 3 delivered the highest effective aggregate throughput at approximately 122.01 tok/s.n_max = 4 was nearly tied in aggregate throughput at approximately 121.00 tok/s, but required substantially more draft work and reduced acceptance to 64.2%.n_max = 2 retained the highest aggregate acceptance rate at 82.2%.n_max = 2, n_max = 3 improved effective aggregate throughput by approximately 5.9%.n_max = 4.n_max = 4.n_max = 4.n_max = 2, reaching 134.3 tok/s.n_max = 2 to 84.8 tok/s at n_max = 4.The results show a clear task-dependent trade-off: predictable output benefits from a longer native MTP draft, while high-entropy natural-language generation generally favors a shorter draft.
Q4_K_M, Q4_K_S, or an Unsloth dynamic quant.The source model is distributed under the Apache License 2.0.
Users should review the upstream nvidia/Qwen3.6-35B-A3B-NVFP4 and Qwen/Qwen3.6-35B-A3B model cards before redistribution or commercial use.