Downloads · 30 days
107
21% of all-time downloads
melcheikh/gemma-4-31B-it-qat-assistant-NVFP4-mse-Blackwell
gemma-4-31B-it-qat-assistant-NVFP4-mse-Blackwell is a text generation model from melcheikh. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
The speculative-decoding draft companion to gemma-4-31B-it-qat-NVFP4-mse-Blackwell, re-quantized with MSE-based weight scaling. It replaces my earlier absmax version
Downloads · 30 days
107
21% of all-time downloads
All-time downloads
512
Public
Parameters
377M
706 MB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors674 MB · 95%
How the weights are stored.
BF16285M · 76%
From the Hugging Face model README
The speculative-decoding draft companion to
gemma-4-31B-it-qat-NVFP4-mse-Blackwell,
re-quantized with MSE-based weight scaling. It replaces my earlier
absmax version
the same way the main model does: per-block scales picked by reconstruction
error instead of the block's largest value, via ModelOpt 0.44.0
(--qformat nvfp4_mse). The MTP handshake layers (pre_projection,
post_projection) stay in high precision, because quantizing those breaks the
draft's connection to the backbone's KV states.
Gemma 4 assistants run on vLLM's MTP path, which exists from vLLM 0.21.0.
google/gemma-4-31B-it-qat-q4_0-unquantized-assistant, 927 MB)
instead. It's small enough that quantization barely matters for VRAM.Measured with the MSE main model as target and the BF16 assistant as draft, on an RTX 6000 Blackwell (300-token generations, temp 0.2): 23.0 tok/s without a draft, 52.1 tok/s with, ~54% token acceptance (78% at position 0).
vllm serve melcheikh/gemma-4-31B-it-qat-NVFP4-mse-Blackwell \
--quantization modelopt_fp4 \
--kv-cache-dtype fp8 \
--speculative-config '{"method": "mtp", "model": "melcheikh/gemma-4-31B-it-qat-assistant-NVFP4-mse-Blackwell", "num_speculative_tokens": 3}'
An earlier version of this card said this draft works on vLLM 0.20.x with
--speculative-model. That was wrong: those flags were removed long before
Gemma 4 support, and 0.20.x has no MTP path for Gemma 4.
Apache 2.0, same as the base model. See the Gemma 4 license.