Gemma-4-31B-it-QAT-NVFP4-GGUF

Gemma 4 31B Instruct - QAT + NVFP4 Hybrid GGUF

QAT-optimized weights preserved at Q4_0, overhead tensors quantized to NVFP4.

What Makes This Different

This is a hybrid quantization of Google official QAT (Quantization-Aware Training) model. Instead of requantizing the Q4_0 weights (which breaks QAT benefits and vision quality), we:

1. Kept all weight tensors at Q4_0 - attention, FFN, embeddings - exactly as Google trained them
2. Quantized only the F32 norm/bias tensors to NVFP4 - these are the overhead tensors (layer norms, RMS norms, etc.)
3. Used Google QAT mmproj - the vision projector trained alongside the QAT model

Why Standard NVFP4 from QAT Breaks Vision

Google QAT model was specifically trained to be resilient to Q4_0 quantization patterns. The weight values learned during QAT compensate for Q4_0 rounding. When you requantize Q4_0 -> F32 -> NVFP4, a second round of quantization error is introduced that QAT training did not account for. Vision tokens flow through the same attention/FFN layers - precision loss disproportionately degrades vision.

How the Hybrid Approach Works

Using llama-quantize --tensor-type-file with --allow-requantize:

llama-quantize --allow-requantize --tensor-type-file keep_q4.txt input.gguf output.gguf NVFP4

The tensor-type-file lists all Q4_0/Q4_K tensors to keep at their current type. When the quantizer sees cur_type == new_type, it copies the tensor data as-is - zero precision loss. Only the remaining F32 tensors are quantized to NVFP4.

Usage

# llama.cpp
llama-server -m gemma-4-31B-it-qat-nvfp4.gguf --mmproj mmproj-gemma-4-31B-it-qat.gguf -ngl 99

Source

• Base model: google/gemma-4-31B-it-qat-q4_0-unquantized
• Quantized with: llama.cpp build 537 (commit d2c6795)
• Chat template: Native Gemma 4 (thinking enabled by default)
• Vision: Full multimodal support via QAT mmproj

Files

License

Apache 2.0 (same as base model)