Gemma-4-12B-it-QAT-NVFP4-GGUF

Gemma 4 12B Instruct QAT — NVFP4 GGUF

NVFP4 (4-bit) GGUF of Google's official QAT release, rebuilt from google/gemma-4-12B-it-qat-q4_0-unquantized and verified on llama.cpp with vision and 256K context at KV Q8_0.

What's in this build

• Quantization: NVFP4 (general.file_type = 39) — NVIDIA Blackwell 4-bit E2M1 values with E4M3 scaling per 16 values plus an FP32 tensor scale.
• Tensor layout: 667 tensors — 328 NVFP4, 338 F32 (norms/biases), and the token embedding matrix kept at Q8_0 (a 262,144 × 3,840 matrix, where 4-bit is a bad trade).
• Params: 11,907,350,576
• BPW: 4.85 (bits per parameter, whole file)
• No hybrid trick: every transformer weight is genuinely NVFP4. Norms stay F32 because quantizing them buys almost nothing and costs accuracy.

Files

The mmproj is shipped at F32/BF16 on purpose. Converting it to F16 breaks vision (see Limitations).

Verified results

Tested on llama.cpp b1868 (4fea119d), RTX 5060 Ti 16 GB, CUDA 13.2:

Vision — OCR test image with five lines including PART-NUMBER: XK-9942, LOT: 7734-B, and 12 + 37 = 49. Output:

GEMMA VISION TEST
PART-NUMBER: XK-9942
LOT: 7734-B
Q4_0 or NVFP4?
12 + 37 = 49

49

All five lines transcribed exactly, arithmetic result correct.

256K context with KV cache at Q8_0 — -c 262144 -ctk q8_0 -ctv q8_0:

• KV cache: K (q8_0) 1088 MiB + V (q8_0) 1088 MiB over 262,144 cells (8 layers), plus 850 MiB SWA cache (5,120 cells, 40 layers)
• Projected 12,953 MiB of 15,174 MiB free device memory → fits with ~2.2 GB headroom
• 257,620-token prompt processed and answered, exit code 0
• Prompt 788.3 t/s, generation 19.5 t/s
• Quantized V cache automatically enables flash attention (required by llama.cpp)

MXFP4 vs NVFP4 on this hardware

Both formats were benchmarked head-to-head on the same machine, same llama.cpp build, same flags, same KV cache setting. Choose based on which half of inference you care about — the result is a genuine trade-off, not a clean win.

Prompt processing (prefill) — NVFP4 wins

Token generation (decode) — MXFP4 wins

Why the split

Both formats take the same native Blackwell FP4 tensor-core path. This build does not privilege one over the other:

const bool use_native_fp4 = blackwell_mma_available(cc) &&
    (src0->type == GGML_TYPE_MXFP4 || src0->type == GGML_TYPE_NVFP4);

The build was compiled with CMAKE_CUDA_ARCHITECTURES=native and ggml-cuda.dll contains sm_120, so the gate is satisfied on RTX 50-series. The difference is therefore file size vs. compute efficiency, not kernel availability:

• NVFP4 is 5.0% larger — 7,219,672,864 B (4.85 BPW) vs 6,879,049,504 B (4.62 BPW) — because its block size is 16 with an E4M3 block scale plus an FP32 tensor scale, where MXFP4 uses block size 32 with a single E8M0 exponent.
• Prefill is compute-bound. NVFP4's finer-grained scaling maps better onto the FP4 tensor cores and the extra bytes do not matter. It wins by up to 17%.
• Decode is memory-bandwidth-bound. Every weight is read per token, so the 5% larger file costs roughly 5-8% of throughput. MXFP4 wins.
• At 250K depth decode becomes attention-bound rather than weight-bandwidth-bound, so the size penalty fades and the two converge.

Bottom line

Sizes to plan VRAM around: MXFP4 6.41 GiB · NVFP4 6.72 GiB. At 256K context with KV Q8_0, the KV cache adds ~2.4 GiB either way.

Usage

# text
llama-cli -m gemma-4-12b-it-qat-nvfp4.gguf -ngl 99 --jinja

# 256K context, KV cache at Q8_0
llama-server -m gemma-4-12b-it-qat-nvfp4.gguf -ngl 99 --jinja \
  -c 262144 -ctk q8_0 -ctv q8_0

# vision / multimodal
llama-mtmd-cli -m gemma-4-12b-it-qat-nvfp4.gguf \
  --mmproj mmproj-gemma-4-12b-it-qat.gguf \
  --image photo.png -ngl 99 --jinja

--jinja is required

Gemma 4's chat template cannot be expressed in llama.cpp's legacy template converter. Without --jinja, llama-mtmd-cli throws an uncaught C++ exception during chat-template setup and dies with silent exit code 0xC0000409 (-1073740791) — no error message, no output. With --jinja the template renders correctly (<|turn>…<turn|>, <|think|>).

Limitations

Do not regenerate the mmproj with a plain F16 conversion. F16 conversion of v.patch_embd.weight (6912×3840) overflows F16 accumulation during the patch-embedding matmul. Observed outcomes on this exact model:

• F16 mmproj → repeated <unused49> tokens, or
• F16 mmproj → silent hallucination (model invents image content instead of reading it)

The F32/BF16 mmproj shipped here is unaffected. This matches upstream llama.cpp issue #24146 (closed not-planned, unfixed), root cause 2: "The patch_embeddings_0 matmul overflows F16 accumulation… forcing F32 accumulation fixes it."

Upstream #24146 root cause 1 (resize not filling the soft-token budget) is also unfixed and was not worked around here; it affects token-count accuracy for images, not correctness of this build's OCR results.

Architecture: Gemma4UnifiedForConditionalGeneration, 48 layers, hidden 3840, FFN 15360, 16 KV heads, sliding-window (1024) + full attention in a 40:8 per-layer pattern (matching the KV cache: 40 SWA layers at 5,120 cells + 8 full layers at 262,144 cells). Native 262,144-token context (gemma4.context_length = 262144). Thinking is enabled by default via the <|think|> token.

Source

• Base model: google/gemma-4-12B-it-qat-q4_0-unquantized
• Converter: llama.cpp convert_hf_to_gguf.py, build b1868 (4fea119d)
• Quantizer: llama-quantize NVFP4 (locally patched build; NVFP4 is not in stock llama.cpp)
• Chat template: native Gemma 4, taken from the source chat_template.jinja

License

Apache 2.0 (same as the base model).