gemma-4-31B-it-assistant-MTP-GGUF

gemma-4-31B-it-assistant — MTP drafter GGUF (Q8_0 + NVFP4)

GGUF builds of google/gemma-4-31B-it-assistant, the official 4-block assistant (MTP) drafter for google/gemma-4-31B-it, quantized for speculative decoding of gemma-4-31B-it GGUF trunks.

The assistant is a small draft head (4 layers, hidden 1024, tied embeddings, 262k vocab) that computes only its own queries and reads the target model's KV cache (attention_k_eq_v, all 4 layers KV-shared), so its per-draft-token cost is a small fraction of a target decode step. The target model verifies every drafted token; the emitted stream is the target's own output.

Files

Both files carry the full 17,336-byte gemma-4 chat template in their GGUF metadata, byte-identical to the source checkpoint's (verified on the exact published bytes — quantization tooling can silently drop chat templates, so this was gated, not assumed).

How they were made

• The official google/gemma-4-31B-it-assistant bf16 checkpoint was converted to an F16 GGUF with a byte-parity gate against the source weights (per-layer scalars byte-equal; output norm exact on all 1024 rows).
• llama-quantize produced the Q8_0 and NVFP4 files from that F16 conversion (no imatrix). NVFP4 is a Blackwell-native 4-bit float format; it was measured on RTX PRO 6000 Blackwell (sm_120). On other hardware, use the Q8_0 file.
• The F16 conversion itself is deliberately not published as a drafter: a float MTP head does not arm the tiyuvta serving engine's gemma speculative route; the quantized builds are the servable artifacts.

Measured results

All numbers are our own measurements of these exact files. Acceptance is protocol-dependent — trunk quantization, sampling, workload shape and verification policy all move it — so every number below states its protocol.

Acceptance / identity / throughput A/B (fixed workload)

Protocol: the serving engine (v0.95.0/v0.96.0 candidates), greedy decoding, single stream, 1× RTX PRO 6000 Blackwell 96 GB; trunk = a Q6_K-class GGUF build of gemma-4-31B-it; draft depth K=5; 12 held-out agent-shaped prompts; interleaved ×5 with one fresh boot per arm per repetition; correctness gate outranks speed — the speculative stream must reproduce plain greedy decode byte-exactly.

The two quantizations are interchangeable on this protocol: acceptance is identical to the count, outputs are byte-identical to plain decode in both arms, and throughput is a wash (~2.03× plain on this cell). What NVFP4 buys is ~100.5 MB on disk/transfer and ~96 MiB resident VRAM.

Serving-config A/B (measured 2026-08-20)

Four fixed greedy probes (256-token budget) through the serving engine running gemma-4-31B-it, minutes apart, Q8_0 vs NVFP4 under the identical serving configuration: outputs byte-identical on 4/4 probes (content and reasoning channels); probe-set acceptance 549/817 = 0.672 (Q8_0) vs 547/823 = 0.665 (NVFP4) — a wash.

Lineage pairing matters

This is the official-lineage head (minted from the bf16 gemma-4-31B-it-assistant checkpoint). In our A/Bs it measured 0.58–0.60 prose acceptance on a trunk built from the official gemma-4-31B-it weights, and 0.28–0.34 when cross-paired with a QAT-lineage trunk. Pair this drafter with trunks derived from the official weights; QAT trunks want the QAT assistant head.

Serving

The serving engine serves gemma-4-31B-it with this drafter attached, and the gemma speculative route arms automatically (K=5; engages on greedy, unconstrained, text-only sessions; speculative output is gated byte-identical to plain decode).

License

apache-2.0, inherited from google/gemma-4-31B-it-assistant; see the Gemma 4 license note linked from the base model card.