gemma-4-31B-it-MXFP4-Quark-RDNA4

Gemma-4-31B-it — MXFP4 (Quark) for RDNA4, 262k window with fp8 KV, native MTP-3 drafter

Release 2026-09-03. Every number below was measured on this artifact on the hardware named, in one gate window. Nothing here is a claim about other stacks or other cards.

MXFP4 (OCP microscaling) quantization of google/gemma-4-31B-it made with AMD Quark, served by the RDNA4 vLLM port capicua25x/vllm-rocm-rdna4 on two Radeon AI PRO R9700 (gfx1201, 32 GB each). License headline: the weights are under the Gemma Terms of Use, a permissive but proprietary license that Google can update and enforce, i.e. revocable in practice; it is not open source. Derivatives carry the same terms and the use-restriction notice (NOTICE). The Apache-2.0 models on this image are Ornith-1.5 and Muse-Glimmer-30B.

Why this model got a quant: the dense 31B is the strongest Gemma-4 for agentic work on this hardware (τ²-bench telecom 0.75 with thinking, versus 0.46 for the 26B-A4B MoE), it keeps Google's MTP draft head (google/gemma-4-31B-it-assistant, served unquantized), and with fp8 KV it runs the full 262,144-token window on two 32 GB cards. At 16 GB of MXFP4 weights per card it also streams tokens faster than a 27B FP8 model on the same bus.

What is quantized

• MXFP4 (fp4 e2m1 weights, group 32, e8m0 shared scales) on all 60 MLP blocks (gate_proj / up_proj / down_proj).
• Kept in bf16: every attention projection (q/k/v/o_proj), the vision tower, lm_head, norms and embeddings.
• Activations: the Quark export declares dynamic MXFP4 input quantization; on RDNA4 the port's MXFP4×fp8 WMMA kernel consumes it (same path as the other models in this collection).
• Export: Quark real-quantized weights, pack_method: reorder, quant_method: quark.

Serving config — what differs from the raw Quark export

The shipped config.json is the raw export plus one fix without which vLLM 0.28 cannot load the model: self_attn.v_proj is added to exclude for the ten attention_k_eq_v global layers (5, 11, … 59). Those layers have no v_proj on disk; Quark's exclude list named only q/k/o, and vLLM's fused qkv_proj requires one scheme for all three shards. (This export already carries the flat global_head_dim: 512 / num_global_key_value_heads: 4 keys the port reads.)

Serving (vLLM on RDNA4 — 2× Radeon AI PRO R9700, TP2)

docker run --rm --network=host --device=/dev/kfd --device=/dev/dri --group-add video --group-add render --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface capicua25x/vllm-rocm-rdna4:0.28.0-rdna4-rc13 \
  serve Capicua25x/gemma-4-31B-it-MXFP4-Quark-RDNA4 --port 8011 --trust-remote-code --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.95 --max-model-len 262144 --kv-cache-dtype fp8 --attention-backend TRITON_ATTN \
  --enable-prefix-caching --max-num-seqs 32 --max-num-batched-tokens 8000 --max-cudagraph-capture-size 128 --skip-mm-profiling \
  --enable-auto-tool-choice --tool-call-parser gemma4 --reasoning-parser gemma4 \
  --speculative-config '{"model":"google/gemma-4-31B-it-assistant","num_speculative_tokens":3}'

• KV sizing is the constraint on 32 GB cards. With bf16 KV one 262k request needs 16.5 GiB per card (11.5 GiB at 131k; the sliding-window layers add a fixed base) and only ~8.7 GiB is free next to the weights and the draft. fp8 KV halves it: the command above boots with a 287,588-token KV pool (1.1× a full 262k request), weights 16.4 GiB per card, ~11 min to ready on a cold cache. The fp8 KV scales are uncalibrated (vLLM defaults); the quality rows below were measured that way.
• Thinking is a chat-template kwarg: "chat_template_kwargs": {"enable_thinking": true}; the gemma4 reasoning parser separates reasoning_content. reasoning_effort is inert on vLLM.
• The MTP drafter is Google's bf16 assistant head (927 MB); vLLM pulls it from the Hub.

Quality (this artifact, this hardware)

Throughput

Bench v3 (fixed prompts, temp 0, MTP-3), tokens/s per user:

Read the two rows as the envelope, not as typical: on the fixed bench prompts the MTP head either agrees with the target on every token (the 6k rows, 4 tokens per step) or on none (the short row, which is therefore the plain-decoding floor). The realistic middle is the live sample under the τ² load (4 requests in flight, thinking on): 156 tok/s aggregate, ~39 tok/s per request, 65 % draft acceptance (2.9 tokens per step), prefill ~1,900 tok/s.

Long context

Needle-in-a-haystack (3 depths per size), prompt tokens as counted by the server, fp8 KV:

Times are cold-prefill latencies to the answer (about 2× the 26B-A4B's: a dense 31B with head-512 global layers); warm turns are interactive.

Credits

Google DeepMind for Gemma 4 and its MTP assistant head; AMD for Quark; the vLLM project. The RDNA4 port, the quant and the measurements are by Capicua25x.