Downloads · 30 days
241
7% of all-time downloads
melcheikh/gemma-4-31B-it-qat-NVFP4-mse-Blackwell
gemma-4-31B-it-qat-NVFP4-mse-Blackwell is a text generation model from melcheikh. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
This is a second pass at quantizing Gemma 4 31B IT to NVFP4. My earlier upload (gemma-4-31B-it-qat-NVFP4-Blackwell) used ModelOpt's default absmax scaling. This one uses MSE-based weight scaling, and I'm publishing it…
Downloads · 30 days
241
7% of all-time downloads
All-time downloads
3.2K
Public
Parameters
16.6B
20.5 GB on disk
Likes
3
Public
Click a slice to open those files.
.safetensors20.4 GB · 100%
How the weights are stored.
U814.6B · 88%
From the Hugging Face model README
This is a second pass at quantizing Gemma 4 31B IT to NVFP4. My earlier upload (gemma-4-31B-it-qat-NVFP4-Blackwell) used ModelOpt's default absmax scaling. This one uses MSE-based weight scaling, and I'm publishing it as a separate repo so nobody's deployment changes under their feet.
With absmax, each 16-weight block gets its scale from the single largest value
in the block. One outlier and the other fifteen weights get crushed toward
zero. That's the accuracy gap NVIDIA called out in their
Nemotron 3 NVFP4 write-up,
and their fix is what I adopted here: per-block scales chosen to minimize
reconstruction error (MSE) plus an FP8 activation scale sweep. In ModelOpt
terms, --qformat nvfp4_mse instead of --qformat nvfp4. Same ModelOpt
release (0.44.0), same calibration set (cnn_dailymail, 512 samples), same
excluded modules (lm_head, vision tower). Calibration takes about 7 minutes
on an RTX 6000 Blackwell instead of 3, because the sweep does more work per batch.
How much it buys you depends on the task. On my own check, extracting figures from a realistic 8-K filing at temp 0.2, this checkpoint and the absmax one tied: 6/6 exact figures, valid JSON, near-identical summaries. NVIDIA's write-up reports the gains on harder benchmarks. I switched because it's the better validated recipe and showed no regression, not because I measured a quality jump.
Do not bake calibrated FP8 KV-cache scales into a Gemma 4 NVFP4 export. I
tried --kv_cache_qformat fp8 and the served model produced degenerate
output, with two different weight quantizations producing byte-identical
garbage, which pins the failure on the KV scales, not the weights. Gemma 4's
mixed head dims (256 local / 512 global) push vLLM onto its Triton attention
backend, and that path misapplies the per-layer checkpoint scales. This
checkpoint ships with no KV scales (kv_cache_quant_algo: null); serve it
with --kv-cache-dtype fp8 and let vLLM do dynamic scaling. Same memory
savings, correct output.
vllm serve melcheikh/gemma-4-31B-it-qat-NVFP4-mse-Blackwell \
--quantization modelopt_fp4 \
--kv-cache-dtype fp8 \
--enforce-eager
Needs a Blackwell GPU for native FP4. Tested on an RTX 6000 Blackwell (96 GB)
and a 24 GB RTX 5090 laptop. On 24 GB you'll want --max-model-len around
16000–20000 or the KV cache won't fit.
Gemma 4 drafts with its MTP assistant model, which needs vLLM 0.21.0 or newer:
vllm serve melcheikh/gemma-4-31B-it-qat-NVFP4-mse-Blackwell \
--quantization modelopt_fp4 \
--kv-cache-dtype fp8 \
--speculative-config '{"method": "mtp", "model": "google/gemma-4-31B-it-qat-q4_0-unquantized-assistant", "num_speculative_tokens": 3}'
Measured on vLLM 0.22.1 with the RTX 6000 Blackwell and this model as target, 300-token generations at temp 0.2: 23.0 tok/s without a draft, 52.1 tok/s with (2.26×, ~54% acceptance). That run used Google's unquantized BF16 assistant (927 MB). Releases before 0.25.0 load the assistant without applying its quant config, so an NVFP4 draft fails there. vLLM 0.25.0 fixed that (#47091), so the NVFP4 assistant should load on current releases, but I haven't re-measured with it yet.
The FP4 weights are packed two per byte, so the tensor shapes are half-size and the parameter auto-count comes out near 17B. The architecture is the full 31B dense model.
Apache 2.0, same as the base model. See the Gemma 4 license.