Downloads · 30 days
134
11% of all-time downloads
williamliao/Gemma-4-26B-A4B-NVFP4-GGUF
Gemma-4-26B-A4B-NVFP4-GGUF is a text generation model from williamliao. Use it when you need the model to write or continue text. It is set up for llama.cpp. The card lists the license as apache-2.0.
GGUF conversion of nvidia/Gemma-4-26B-A4B-NVFP4 for use with llama.cpp.
Downloads · 30 days
134
11% of all-time downloads
All-time downloads
1.2K
Public
Repo size
17.7 GB
Likes
0
Public
Click a slice to open those files.
.gguf17.7 GB · 100%
From the Hugging Face model README
GGUF conversion of nvidia/Gemma-4-26B-A4B-NVFP4 for use with llama.cpp.
This is an NVFP4-quantized standalone Gemma 4 target model.
It can run by itself. For the speculative-decoding results below, it was paired with a compatible Gemma 4 26B A4B Assistant/MTP draft model.
nvidia/Gemma-4-26B-A4B-NVFP4google/gemma-4-26B-A4B-itnvfp4_experts_onlyThe upstream NVIDIA checkpoint was quantized with NVIDIA Model Optimizer v0.43.0. NVIDIA describes it as using an nvfp4_experts_only recipe, so not every tensor in the checkpoint is expected to use NVFP4.
This repository contains the complete target language model. The Assistant/MTP draft model used for speculative decoding is separate.
A recent version of llama.cpp with Gemma 4 MoE, NVFP4, and Gemma 4 Assistant/MTP support is required.
Tested with:
--spec-type draft-mtpOlder llama.cpp builds may fail to recognize the nvfp4 tensor type, load the Gemma 4 Assistant architecture, or initialize the shared KV-cache layout correctly.
Performance varies with the llama.cpp build, target and draft quantizations, GPU split, context size, KV-cache format, prompt, sampling settings, and other runtime options.
Multimodal inference requires a compatible projector and runtime support. The text-only benchmark below does not test image input.
Replace the draft filename below with your compatible Gemma 4 26B A4B Assistant GGUF.
llama-server \
-m Gemma-4-26B-A4B-NVFP4.gguf \
-md gemma-4-26B-A4B-it-assistant.gguf \
--spec-type draft-mtp \
--spec-draft-n-max 4 \
--flash-attn on \
-ngl 99
llama-cli \
-m Gemma-4-26B-A4B-NVFP4.gguf \
-md gemma-4-26B-A4B-it-assistant.gguf \
--spec-type draft-mtp \
--spec-draft-n-max 4 \
--flash-attn on \
-ngl 99
llama-server \
-m Gemma-4-26B-A4B-NVFP4.gguf \
-md gemma-4-26B-A4B-it-assistant.gguf \
--spec-type draft-mtp \
--spec-draft-n-max 4 \
--split-mode layer \
--tensor-split 1,1 \
--flash-attn on \
-ngl 99
The target and Assistant models must use compatible tokenizer, vocabulary, and Gemma 4 architecture assumptions.
An even tensor split is only a starting point. The best split depends on GPU speed, available VRAM, PCIe topology, context length, and placement of the target model, draft model, and KV cache.
Based on the mixed-task Assistant benchmark below:
n_max = 4 gave the best aggregate throughput and shortest wall time.n_max = 2 gave the highest aggregate acceptance rate and remained preferable for several high-entropy tasks.n_max = 5 was nearly tied with n_max = 4 overall and achieved the highest speeds for repeated patterns, JSON, code completion, and stepwise mathematics.n_max = 4 to n_max = 5 added substantially more draft work but did not improve aggregate throughput.Recommended general-purpose setting:
--spec-draft-n-max 4
For role-play, creative writing, open-ended explanations, conversational output, or long code review:
--spec-draft-n-max 2
For JSON, repeated patterns, deterministic code completion, and other highly predictable output:
--spec-draft-n-max 5
Converted from the original Hugging Face ModelOpt checkpoint using a recent convert_hf_to_gguf.py from llama.cpp:
python convert_hf_to_gguf.py \
/path/to/Gemma-4-26B-A4B-NVFP4 \
--outfile Gemma-4-26B-A4B-NVFP4.gguf \
--outtype auto \
--verbose
Do not pass a conventional GGUF quantization type such as q4_k_m when the goal is to preserve the original NVFP4 tensors. The conversion repackages the ModelOpt checkpoint into GGUF rather than requantizing it into a K-quant.
The compatible Assistant model must be converted separately. No target-model weights were retrained by this conversion.
The converted GGUF may be larger than the original Hugging Face ModelOpt checkpoint.
The conversion does not copy the original safetensors byte-for-byte. NVFP4 tensors are repacked into GGML-compatible layouts, and GGUF stores the associated scale metadata required by llama.cpp. Higher-precision tensors are also retained where required by the source checkpoint.
A larger GGUF does not mean the model was converted to FP16 or to a conventional Q4 format.
Windows reports file sizes using binary units even though File Explorer labels them as GB. A file shown with a larger decimal-GB value on Hugging Face can therefore appear smaller in Windows.
File size is not the same as runtime VRAM usage. Additional memory is needed for:
The converted target can be checked with llama-gguf:
.\llama-gguf.exe E:\HF_MODELS\Gemma-4-26B-A4B-NVFP4.gguf r
A successful conversion should show NVFP4 expert tensors together with their associated scale tensors. Other tensors may remain in BF16, FP16, FP8, or other source-compatible types because NVIDIA used an experts-only NVFP4 recipe.
Example:
tensor: name = blk.0.ffn_*.weight, type = nvfp4
tensor: name = blk.0.ffn_*.scale, type = f32
tensor: name = blk.0.ffn_*.input_scale, type = f32
This confirms that NVFP4 tensors were preserved and that the output is not an empty metadata-only GGUF.
Benchmarked using mtp-bench with a compatible Gemma 4 Assistant model and:
--spec-type draft-mtp
n_max | Draft acceptance | Predicted tokens | Draft tokens | Accepted tokens | Wall time | Effective batch throughput |
|---|---|---|---|---|---|---|
| 2 | 80.6% | 2,135 | 1,622 | 1,307 | 16.77 s | 127.31 tok/s |
| 3 | 74.0% | 2,129 | 1,969 | 1,456 | 15.58 s | 136.65 tok/s |
| 4 | 68.1% | 2,116 | 2,261 | 1,540 | 14.76 s | 143.36 tok/s |
| 5 | 60.7% | 2,116 | 2,606 | 1,582 | 14.81 s | 142.88 tok/s |
Effective batch throughput is calculated as total predicted tokens divided by total wall time.
Per-request tok/s can be higher because aggregate wall time may include prompt processing, request transitions, and benchmark overhead.
The predicted-token totals differ slightly because some generated responses terminated at different lengths. The n_max = 4 and n_max = 5 runs both generated 2,116 tokens and can be compared directly.
These tests do not include a non-speculative baseline and therefore should not be interpreted as direct speedup ratios over standard decoding.
n_max = 2code_python pred= 192 draft= 139 acc= 121 rate=0.871 tok/s=144.7
code_cpp pred= 192 draft= 148 acc= 117 rate=0.790 tok/s=138.9
explain_concept pred= 192 draft= 158 acc= 111 rate=0.703 tok/s=128.2
summarize pred= 52 draft= 44 acc= 30 rate=0.682 tok/s=125.8
qa_factual pred= 192 draft= 166 acc= 108 rate=0.651 tok/s=125.1
translation pred= 129 draft= 102 acc= 77 rate=0.755 tok/s=135.5
creative_short pred= 34 draft= 32 acc= 17 rate=0.531 tok/s=114.0
stepwise_math pred= 192 draft= 137 acc= 122 rate=0.890 tok/s=145.5
json_output pred= 192 draft= 128 acc= 127 rate=0.992 tok/s=158.4
long_reasoning pred= 192 draft= 142 acc= 119 rate=0.838 tok/s=140.9
repeat_pattern pred= 192 draft= 127 acc= 127 rate=1.000 tok/s=159.1
code_completion pred= 192 draft= 132 acc= 124 rate=0.939 tok/s=152.5
long_code_review pred= 192 draft= 167 acc= 107 rate=0.641 tok/s=118.0
Aggregate:
requests: 13
predicted tokens: 2135
draft tokens: 1622
accepted tokens: 1307
acceptance rate: 0.8058
total wall time: 16.77 s
n_max = 3code_python pred= 192 draft= 162 acc= 137 rate=0.846 tok/s=167.6
code_cpp pred= 192 draft= 179 acc= 131 rate=0.732 tok/s=150.7
explain_concept pred= 192 draft= 209 acc= 120 rate=0.574 tok/s=127.7
summarize pred= 52 draft= 54 acc= 33 rate=0.611 tok/s=128.2
qa_factual pred= 192 draft= 207 acc= 121 rate=0.585 tok/s=130.1
translation pred= 123 draft= 123 acc= 82 rate=0.667 tok/s=141.6
creative_short pred= 34 draft= 39 acc= 21 rate=0.538 tok/s=122.6
stepwise_math pred= 192 draft= 159 acc= 137 rate=0.862 tok/s=166.7
json_output pred= 192 draft= 146 acc= 142 rate=0.973 tok/s=181.3
long_reasoning pred= 192 draft= 188 acc= 128 rate=0.681 tok/s=143.5
repeat_pattern pred= 192 draft= 143 acc= 143 rate=1.000 tok/s=188.4
code_completion pred= 192 draft= 151 acc= 140 rate=0.927 tok/s=175.9
long_code_review pred= 192 draft= 209 acc= 121 rate=0.579 tok/s=125.9
Aggregate:
requests: 13
predicted tokens: 2129
draft tokens: 1969
accepted tokens: 1456
acceptance rate: 0.7395
total wall time: 15.58 s
n_max = 4code_python pred= 192 draft= 176 acc= 146 rate=0.830 tok/s=179.8
code_cpp pred= 192 draft= 201 acc= 140 rate=0.697 tok/s=162.6
explain_concept pred= 192 draft= 277 acc= 120 rate=0.433 tok/s=116.2
summarize pred= 52 draft= 60 acc= 38 rate=0.633 tok/s=146.9
qa_factual pred= 192 draft= 250 acc= 128 rate=0.512 tok/s=132.7
translation pred= 110 draft= 120 acc= 82 rate=0.683 tok/s=158.2
creative_short pred= 34 draft= 48 acc= 23 rate=0.479 tok/s=121.9
stepwise_math pred= 192 draft= 176 acc= 147 rate=0.835 tok/s=187.2
json_output pred= 192 draft= 160 acc= 151 rate=0.944 tok/s=203.1
long_reasoning pred= 192 draft= 204 acc= 140 rate=0.686 tok/s=159.4
repeat_pattern pred= 192 draft= 153 acc= 152 rate=0.994 tok/s=211.5
code_completion pred= 192 draft= 164 acc= 150 rate=0.915 tok/s=199.2
long_code_review pred= 192 draft= 272 acc= 123 rate=0.452 tok/s=119.7
Aggregate:
requests: 13
predicted tokens: 2116
draft tokens: 2261
accepted tokens: 1540
acceptance rate: 0.6811
total wall time: 14.76 s
n_max = 5code_python pred= 192 draft= 205 acc= 150 rate=0.732 tok/s=182.1
code_cpp pred= 192 draft= 240 acc= 143 rate=0.596 tok/s=158.5
explain_concept pred= 192 draft= 340 acc= 121 rate=0.356 tok/s=109.0
summarize pred= 52 draft= 80 acc= 37 rate=0.463 tok/s=128.5
qa_factual pred= 192 draft= 279 acc= 135 rate=0.484 tok/s=137.3
translation pred= 110 draft= 150 acc= 81 rate=0.540 tok/s=146.0
creative_short pred= 34 draft= 60 acc= 23 rate=0.383 tok/s=111.1
stepwise_math pred= 192 draft= 189 acc= 153 rate=0.809 tok/s=196.7
json_output pred= 192 draft= 172 acc= 156 rate=0.907 tok/s=212.1
long_reasoning pred= 192 draft= 225 acc= 145 rate=0.644 tok/s=163.1
repeat_pattern pred= 192 draft= 159 acc= 159 rate=1.000 tok/s=234.8
code_completion pred= 192 draft= 178 acc= 155 rate=0.871 tok/s=208.2
long_code_review pred= 192 draft= 329 acc= 124 rate=0.377 tok/s=111.2
Aggregate:
requests: 13
predicted tokens: 2116
draft tokens: 2606
accepted tokens: 1582
acceptance rate: 0.6071
total wall time: 14.81 s
A separate DFlash draft was also tested at n_max = 2:
code_python pred= 192 draft= 148 acc= 117 rate=0.790 tok/s=113.2
code_cpp pred= 192 draft= 164 acc= 108 rate=0.658 tok/s=104.3
explain_concept pred= 192 draft= 204 acc= 88 rate=0.431 tok/s=84.1
summarize pred= 51 draft= 46 acc= 28 rate=0.609 tok/s=98.0
qa_factual pred= 192 draft= 179 acc= 101 rate=0.564 tok/s=95.9
translation pred= 70 draft= 78 acc= 31 rate=0.397 tok/s=80.7
creative_short pred= 34 draft= 40 acc= 14 rate=0.350 tok/s=76.1
stepwise_math pred= 192 draft= 151 acc= 115 rate=0.762 tok/s=113.1
json_output pred= 192 draft= 134 acc= 124 rate=0.925 tok/s=126.4
long_reasoning pred= 192 draft= 163 acc= 108 rate=0.663 tok/s=101.8
repeat_pattern pred= 192 draft= 132 acc= 125 rate=0.947 tok/s=129.2
code_completion pred= 192 draft= 137 acc= 122 rate=0.890 tok/s=123.9
long_code_review pred= 192 draft= 193 acc= 94 rate=0.487 tok/s=86.7
Aggregate:
requests: 13
predicted tokens: 2075
draft tokens: 1769
accepted tokens: 1175
acceptance rate: 0.6642
total wall time: 22.10 s
effective throughput: 93.89 tok/s
| Draft method | Best tested setting | Draft acceptance | Effective batch throughput |
|---|---|---|---|
| DFlash | n_max = 2 | 66.4% | 93.89 tok/s |
| Assistant/MTP | n_max = 4 | 68.1% | 143.36 tok/s |
In this benchmark, Assistant/MTP at n_max = 4 delivered approximately 52.7% higher effective aggregate throughput than the tested DFlash n_max = 2 run.
This comparison is specific to the tested draft checkpoints, quantizations, hardware, prompts, and runtime configuration. It should not be interpreted as a universal comparison between the Assistant and DFlash methods.
n_max = 4 was the best mixed-task setting, reaching approximately 143.36 tok/s aggregate throughput.n_max = 5 was effectively tied overall at 142.88 tok/s, but used 15.3% more draft tokens than n_max = 4.n_max = 4 to n_max = 5 increased accepted draft tokens by only 2.7%, indicating sharply diminishing returns.n_max = 5 reached 234.8 tok/s on repeated-pattern generation.n_max = 5.n_max = 5.n_max = 5.n_max = 5.The benchmark shows a clear task-dependent trade-off: structured and predictable output benefits from deeper drafting, while high-entropy natural-language generation tends to favor shorter drafts.
Q4_K_M, Q4_K_S, or an Unsloth dynamic quant.google/gemma-4-26B-A4B-itnvidia/Gemma-4-26B-A4B-NVFP4The source checkpoint is distributed under the Apache License 2.0.
Users should review the upstream nvidia/Gemma-4-26B-A4B-NVFP4 and google/gemma-4-26B-A4B-it model cards and applicable terms before redistribution or commercial use.