Gemma-4-12B-it-QAT-MXFP4-GGUF

Gemma 4 12B Instruct QAT — MXFP4 GGUF

Genuine MXFP4 (4-bit) GGUF of Google's official QAT release, rebuilt from google/gemma-4-12B-it-qat-q4_0-unquantized and verified on llama.cpp with vision, 128K and 256K context, and KV cache at Q8_0.

What's in this build

• Quantization: MXFP4 (general.file_type = 42, ggml type 39) — OCP microscaled FP4: 4-bit E2M1 values with E8M0 power-of-two scaling per 32 values.
• Tensor layout: 667 tensors — 328 MXFP4, 338 F32 (norms/biases), and the token embedding matrix kept at Q8_0 (a 262,144 × 3,840 matrix, where 4-bit is a bad trade).
• Params: 11,907,350,576
• BPW: 4.62 (bits per parameter, whole file)
• No hybrid trick: every transformer weight is genuinely MXFP4. Norms stay F32 because quantizing them buys almost nothing and costs accuracy.
• Chat template: Google's canonical Gemma 4 template, Published: 2026-07-09 — the fix for tool-calling loops, turn closures, and thinking content-ordering.

Files

The mmproj is shipped at F32/BF16 on purpose. Converting it to F16 breaks vision (see Limitations).

Verified results

All numbers measured on llama.cpp b1868 (4fea119d), RTX 5060 Ti 16 GB (Blackwell, SM120a), CUDA 13.2, host CPU AMD Ryzen 7 5700X (8 cores / 16 threads), 64 GB RAM.

Context length comparison — the headline

Both runs processed a prompt sized with the model's own tokenizer, with a unique retrieval fact buried at ~60% depth. Both answered correctly.

Doubling the window costs 38% of prompt throughput and 28% of generation throughput. 128K is the practical sweet spot on this card: it is both faster and leaves much more VRAM headroom. Use 256K only when you actually need the window.

Vision

OCR test image with five lines. Output:

GEMMA VISION TEST
PART-NUMBER: XK-9942
LOT: 7734-B
Q4_0 or MXFP4?
12 + 37 = 49

12 + 37 = 49 — correct

All five lines transcribed exactly, arithmetic confirmed, exit code 0. Run with llama-mtmd-cli + --jinja.

128K context, KV cache Q8_0

-c 131072 -ctk q8_0 -ctv q8_0, 127,740-token prompt.

+3.0% prompt, generation within noise. Exit code 0, retrieval fact recovered.

256K context, KV cache Q8_0

-c 262144 -ctk q8_0 -ctv q8_0, 255,396-token prompt.

+1.5% prompt, 0% generation. Exit code 0, retrieval fact recovered.

VRAM breakdown

At full 256K with KV Q8_0 (CUDA0, 16,283 MiB total):

• KV cache at 256K: 262,144 cells non-SWA + 1,536 cells SWA, both Q8_0
• 49/49 layers offloaded to GPU
• Flash attention auto-enabled (llama.cpp requires it for a quantized V cache)

At 128K the context allocation halves to roughly 1.2 GB, leaving well over 6 GB free.

Performance tuning on this hardware

Three sweeps were run with llama-bench. The short version: this workload is attention-bound, so most published tuning advice does not apply, and what helps at short context stops helping at long context.

ubatch at short context (pp8192 / tg128)

ubatch 1024 is the peak; 2048 and above regress. This contradicts the widespread advice to push ubatch to 4096 for maximum prompt speed — that advice does not hold on this GPU. Generation is completely flat across ubatch (~51.7 t/s), so there is no generation penalty for large batches here, unlike setups where the model exceeds available memory.

ubatch at real context depth

At depth the ubatch spread collapses to ~1% at 128K and ~0.5% at 256K — statistical noise. Going from short context to 250K depth drops prompt processing from ~2,870 to ~437 t/s (−85%) and generation from 51.7 to ~21 t/s (−59%). The context length, not the batch size, is what matters.

Threads, pure generation (-p 0, tg128)

A 1.7% spread from 2 to 16 threads. With every layer on the GPU, the CPU only handles sampling and dispatch, so thread count is irrelevant to generation speed. -tb/--threads-batch affects batch and prompt processing only (per llama.cpp's own help text) and cannot influence the decode loop at all.

What actually matters

• Context length dominates everything. 128K is 1.6x the prompt speed and 1.4x the generation speed of 256K. Choose the smallest window that fits your task.
• ubatch 1024 is worth +2.4% prompt speed at short context, +3.0% at 128K, +1.5% at 256K. Small but free.
• -fa is effectively always on — a quantized V cache requires it, so there is nothing to tune.
• Threads and -tb are not levers on a fully GPU-offloaded model.

MXFP4 vs NVFP4 on this hardware

Both formats were benchmarked head-to-head on the same machine, same llama.cpp build, same flags, same KV cache setting. Choose based on which half of inference you care about — the result is a genuine trade-off, not a clean win.

Prompt processing (prefill) — NVFP4 wins

Token generation (decode) — MXFP4 wins

Why the split

Both formats take the same native Blackwell FP4 tensor-core path. This build does not privilege one over the other:

const bool use_native_fp4 = blackwell_mma_available(cc) &&
    (src0->type == GGML_TYPE_MXFP4 || src0->type == GGML_TYPE_NVFP4);

The build was compiled with CMAKE_CUDA_ARCHITECTURES=native and ggml-cuda.dll contains sm_120, so the gate is satisfied on RTX 50-series. The difference is therefore file size vs. compute efficiency, not kernel availability:

• NVFP4 is 5.0% larger — 7,219,672,864 B (4.85 BPW) vs 6,879,049,504 B (4.62 BPW) — because its block size is 16 with an E4M3 block scale plus an FP32 tensor scale, where MXFP4 uses block size 32 with a single E8M0 exponent.
• Prefill is compute-bound. NVFP4's finer-grained scaling maps better onto the FP4 tensor cores and the extra bytes do not matter. It wins by up to 17%.
• Decode is memory-bandwidth-bound. Every weight is read per token, so the 5% larger file costs roughly 5-8% of throughput. MXFP4 wins.
• At 250K depth decode becomes attention-bound rather than weight-bandwidth-bound, so the size penalty fades and the two converge.

Bottom line

Sizes to plan VRAM around: MXFP4 6.41 GiB · NVFP4 6.72 GiB. At 256K context with KV Q8_0, the KV cache adds ~2.4 GiB either way.

Usage

# text — 128K, the recommended default on a 16 GB card
llama-cli -m gemma-4-12b-it-qat-mxfp4.gguf \
  -ngl 99 -c 131072 -ctk q8_0 -ctv q8_0 \
  -ub 1024 -b 2048 -fa on --jinja

# text — full 256K window when you need it
llama-cli -m gemma-4-12b-it-qat-mxfp4.gguf \
  -ngl 99 -c 262144 -ctk q8_0 -ctv q8_0 \
  -ub 1024 -b 2048 -fa on --jinja

# vision — the mmproj and --jinja are both mandatory
llama-mtmd-cli -m gemma-4-12b-it-qat-mxfp4.gguf \
  --mmproj mmproj-gemma-4-12b-it-qat.gguf \
  --image photo.jpg -ngl 99 --jinja

--jinja is not optional for Gemma 4. Without it the template goes through llama.cpp's legacy converter, throws, and the process dies with 0xC0000409 right after init_audio().

Limitations

• Vision requires the F32/BF16 mmproj. An F16 mmproj (v.patch_embd.weight at 6912×3840) overflows F16 accumulation and produces <unused49> spam or silent hallucination. Upstream llama.cpp issue #24146, closed as not-planned. The mmproj in this repo is Google's official file, not a regenerated one.
• Q4_0 remains the format Google trained this model for. This is a QAT model, so Q4_0 weights carry the training-time quantization alignment. MXFP4 is a genuinely different 4-bit grid; it was chosen here for Blackwell FP4 hardware support and broader portability, not because it beats Q4_0 on quality. If you want the QAT alignment exactly, use Google's official gemma-4-12B-it-qat-q4_0-gguf.
• Long context is slow by nature. Generation drops to ~28 t/s at 128K and ~20 t/s at 256K. This is attention cost over a large KV cache, not a defect of this quantization — the same effect appears on the NVFP4 sibling and on unquantized builds.
• Greedy decoding can loop on long repetitive prompts. With --temp 0 on the 128K test, the model entered a repetition loop ("Scanning further..." indefinitely) instead of answering. At --temp 0.3 --top-p 0.9 it answered correctly on the first try. If you see looping on very long inputs, add a little sampling entropy or a repetition penalty; this is model behaviour, not a quantization artifact.
• This artifact replaces an earlier mislabeled build. The previous file in this repo was a Q4_0 hybrid (328 Q4_0 + 338 F32 + 1 Q4_K) under an MXFP4 name, carrying a stale pre-2026-07-09 chat template. It is preserved in git history at c50ecc6cd3ec. If you downloaded before 2026-10-21, re-download.

Source

• Base model: google/gemma-4-12B-it-qat-q4_0-unquantized
• Pipeline: bf16 safetensors → F16 GGUF (convert_hf_to_gguf.py) → MXFP4 (llama-quantize)
• Template source: chat_template.jinja from the base repo (canonical 2026-07-09)
• mmproj: google/gemma-4-12B-it-qat-q4_0-gguf, shipped unmodified
• Sibling: FreedomAISVR/Gemma-4-12B-it-QAT-NVFP4-GGUF — same model in NVFP4 (4.85 BPW, 328 NVFP4 tensors), for RTX 50-series E4M3-scaled FP4