Gemma-4-31B-it-ROCmFP4-GGUF

Gemma-4-31B-it — ROCmFP4 / ROCmFPX GGUF

First ROCmFP4/ROCmFPX quantization of google/gemma-4-31B-it. Four quants, built from the Hub BF16 GGUF (no re-conversion from safetensors), each individually verified — text and vision — on real hardware.

Built for AMD Strix Halo (gfx1151) — Ryzen AI MAX+ 395, 128 GB unified memory, ROCm 7.2.4 — using the ROCmFPX llama.cpp fork, which adds AMD-native FP4/FP8 tensor types that mainline llama.cpp does not have.

⚠️ These files need a ROCmFPX-capable llama.cpp build. They will not load in stock llama.cpp / Ollama / LM Studio — Q4_0_ROCMFP4_* and Q*_0_ROCMFPX* are not mainline types.

Variants — all four in this repo

mmproj-BF16.gguf (vision projector) is included — required for image input.

Which to pick: the 4-bit (102) — 21.67 t/s with the drafter, 1.46× the fastest 8-bit at 55% of the size, with identical correctness and vision results. If you want an 8-bit, take the AGENT variant Q8_0_ROCMFPX_AGENT (115), not the plain one: with speculation on it is both faster (15.34 vs 14.86) and accepts more drafts (0.619 vs 0.603), because the AGENT recipe holds more attention-output tensors at higher precision — which is exactly what raises draft acceptance. (Without a drafter the ordering reverses and plain Q8 wins; the AGENT recipe only pays when you are actually speculating.)

Measured — not estimated

Hardware: AMD Ryzen AI MAX+ 395 (Strix Halo, gfx1151), 128 GB unified, ROCm 7.2.4. Load: -ngl 999 -c 4096 -fa on -fit off. Decode = 300 tokens, warm-up discarded, median of 3.

With the drafter (n-max 5), correctness re-checked under the shipping config:

Without the drafter (vision path, -fa off):

Spreads ≤1.003 with nothing else running — clean numbers, not contended ones. Speculative decoding is output-preserving (the target verifies every drafted token) and the correctness answers are identical with and without it.

Why you can trust the no-drafter numbers

This model is conventionally dense, so weights stream once per token and t/s × file_GB gives effective memory bandwidth. Against Strix Halo's ~256 GB/s peak, all four land at 81–83% — the hardware ceiling, so nothing is spilling to CPU or silently misconfigured. (With MTP the model verifies several tokens per weight read, which is exactly how it exceeds that ceiling.)

Vision was tested with a 512² four-colour quadrant image, asking which colour is where. A solid-colour square would pass trivially and proves nothing. Use -fa off for image input. ⚠️ Give vision prompts max_tokens: 1024; at 256 the model spends the budget thinking and returns finish_reason: length with empty content, which looks like a failure and is not.

⚠️ If the MTP draft command crashes on your build

The separate-model draft-mtp path has known bugs in the legacy charlie12345/ROCmFPX line — reported on Windows 11 / gfx1151 / HIP SDK 7.2 (five stacked bugs, ending in an h-row width mismatch in the draft's embedding buffer).

Fix: build the official repo instead — no patches needed.

git clone https://github.com/ROCmFPX/ROCmFPX.git

The MTP path was reworked there (unified n_embd_out row widths, t_h_nextn reset in llm_graph_result::reset(), ctx_other wired centrally), which covers the whole chain. Verified by @beeugene, who tracked this down and filed the fix — 24–31 tok/s, coherent output, tool calling working. Legacy-line patch: PR #109.

Linux builds on the legacy line are not known to be affected.

⚡ Speculative decoding (MTP) — supported, and worth 1.84×

This model has an official drafter, google/gemma-4-31B-it-assistant. It is included in this repo as mtp-gemma-4-31B-it-Q8_0.gguf (491 MB) so the numbers above are reproducible.

Recommended text command — this is what produced 21.67 t/s:

llama-server -m gemma-4-31B-it-Q4_0_ROCMFP4_COHERENT.gguf \
  --spec-type draft-mtp --model-draft mtp-gemma-4-31B-it-Q8_0.gguf \
  --spec-draft-ngl 999 --spec-draft-n-max 5 \
  -ngl 999 -c 4096 -fa on -fit off

Measured tuning (median of 3 each, code prompt, idle box) — don't guess these:

Draft acceptance at the recommended config is 0.649, mean accepted length 3.18. Acceptance is workload-dependent: code drafts accept far better than prose, so treat 21.67 as a code-workload figure and expect less on free-form text.

⛔ MTP and vision cannot be used together

Enabling the drafter and passing an image crashes the server:

process: missing MTP boundary for seq_id=2 pos=132
srv update_slots: failed to process speculative batch
server-context.cpp:3202: fatal error

Image embeddings are injected outside the normal token path and the speculative batch loses its boundary (upstream llama.cpp PR #20277). For vision, drop the --spec-type flags and use -fa off — that path is fully verified (see below) and runs at 11.81 t/s.

Verification

Each artifact was checked individually:

• Size vs --dry-run projection — constant ~15.3 MiB GGUF-header delta across all four
• token_embd audited by exact tensor name (not substring — output.weight false-matches blk.N.attn_output.weight)
• Loaded, 3/3 correctness, real-content vision test, decode median of 3 with spread reported

  514687104  mtp-gemma-4-31B-it-Q8_0.gguf   (drafter — needed for the headline speed)
17651002048  gemma-4-31B-it-Q4_0_ROCMFP4_COHERENT.gguf
28594301632  gemma-4-31B-it-Q6_0_ROCMFPX_AGENT.gguf
31720467136  gemma-4-31B-it-Q8_0_ROCMFPX.gguf
32252046016  gemma-4-31B-it-Q8_0_ROCMFPX_AGENT.gguf

⚠️ Note for anyone quantizing this model themselves

gemma-4-31B-it has tie_word_embeddings = true — there is no output.weight tensor. --output-tensor-type is a silent no-op; --token-embedding-type is the only flag that actually protects the head:

llama-quantize --output-tensor-type q6_K --token-embedding-type q6_K \
  gemma-4-31B-it-BF16.gguf out.gguf Q4_0_ROCMFP4_COHERENT 16

Verify by reading the finished file's token_embd.weight type — do not assume it worked because the flag was accepted.

Credits

• Base model: google/gemma-4-31B-it — Gemma Terms of Use
• Quantization stack: ROCmFPX