Gemma-4-E2B-it-ROCmFP4-GGUF

Gemma-4-E2B-it — ROCmFP4 / ROCmFPX GGUF

First ROCmFP4/ROCmFPX quantization of google/gemma-4-E2B-it. Four quants, built from the Hub BF16 GGUF (no re-conversion from safetensors), each individually verified — text and vision — on real hardware.

Built for AMD Strix Halo (gfx1151) — Ryzen AI MAX+ 395, 128 GB unified memory — using the ROCmFPX llama.cpp fork, which adds AMD-native FP4/FP8 tensor types that mainline llama.cpp does not have.

⚠️ These files need a ROCmFPX-capable llama.cpp build. They will not load in stock llama.cpp / Ollama / LM Studio — Q4_0_ROCMFP4_* and Q*_0_ROCMFPX* are not mainline types.

Variants — all four in this repo

⚠️ One sample in this table is an outlier (115: 55.05 against a ~68 baseline). The box was otherwise idle and every other sample sits inside a 1.01x band, so this is a transient, not a bimodal model. The median is reported precisely because it is robust to it, and all five raw samples are shown rather than hidden behind a summary.

* The two 8-bit builds were measured while the same machine was quantizing another model. Their spreads (11% and 20%) show that contention, so treat those two as lower bounds — uncontended single-shot observations reached 56.4 and 57.5 t/s. The 4-bit and 6-bit figures were taken clean (spread under 0.4%) and need no such caveat. We would rather publish a conservative number than a flattering one.

mmproj-BF16.gguf (vision projector) is included — you need it for image input.

Which to pick: the 4-bit (102). It is the smallest and the fastest by a wide margin, and it answered every correctness and vision test identically to the 8-bits. Take an 8-bit only if you want maximum fidelity for its own sake.

⚠️ Why the 4-bit is 5.76 BPW, not ~4.5

E2B is a MatFormer / per-layer-embedding model: a large fraction of its parameters live in per_layer_token_embd and related tensors that are not driven down to 4 bits. So the 4-bit build is only 1.46× smaller than the 8-bit here, where a conventional dense model would be ~1.8×. This is expected for this architecture, not a defect — the quantized layers really are FP4. One tensor, per_layer_model_proj.weight [1536, 8960] (~26 MiB), stays BF16 in all four builds.

Measured — not estimated

Hardware: AMD Ryzen AI MAX+ 395 (Strix Halo, gfx1151), 128 GB unified. Load: -ngl 999 -c 4096 -fa on -fit off. Decode = 300 tokens, warm-up discarded, median of 3.

A note on the usual bandwidth sanity check

For a conventional dense model you can sanity-check a decode figure with t/s × file_GB against the platform's peak memory bandwidth (~256 GB/s here) — anything well under ~80% suggests something is misconfigured. That check does not apply to this model. E2B's per-layer-embedding design means not every byte of the file is read per token, so the arithmetic returns ~99–102% of "peak" and would happily exceed it. Do not read those percentages as validation for MatFormer/PLE (or MoE) architectures.

Vision was tested with a four-colour quadrant image and the model had to say which colour was where — a solid-colour square would pass trivially and proves nothing. Use -fa off for image input.

⚠️ Speculative decoding (MTP): it works, but it is slower — don't use it

Earlier revisions of this card said MTP "does not currently work". That was wrong, and the reason it was wrong is worth stating: it was tested on one machine whose llama.cpp build never wired the MTP source, and the failure was then explained by a model-width theory that happened to fit. On a build that does wire it (ROCmFPX-2809dc5) the draft head loads and drafts correctly.

It is still not worth enabling here:

⚠️ This repo does not ship the draft head (an earlier version of this card wrongly said it did). If you want to reproduce the row above, take MTP/mtp-gemma-4-E2B-it-Q8_0.gguf from unsloth/gemma-4-E2B-it-GGUF. We do not ship it here because it makes this model slower.

0.94x — a net loss at 59% acceptance. Speculation pays inversely to how fast the target already is: on our Gemma-4-31B build the same technique is worth 1.84x at 11.8 t/s, on the 26B-A4B ~1.03-1.15x at ~54 t/s, and here at ~94 t/s it costs more than it returns. The drafter's per-step overhead is roughly fixed; the target's per-token cost is not.

⭐ Acceptance does not predict speedup. A drafter can be right most of the time and still lose.

Verification

Each artifact was checked individually, not sampled:

• Size vs --dry-run projection — constant ~15.8 MB GGUF-header delta on all four (truncation check)
• token_embd audited by exact tensor name
• Loaded, 3/3 correctness, vision content test
• Decode median of 3 with the spread reported

Exact byte sizes:

3360161632  gemma-4-E2B-it-Q4_0_ROCMFP4_COHERENT.gguf
4707581792  gemma-4-E2B-it-Q6_0_ROCMFPX_AGENT.gguf
4908687200  gemma-4-E2B-it-Q8_0_ROCMFPX.gguf
4941201248  gemma-4-E2B-it-Q8_0_ROCMFPX_AGENT.gguf

⚠️ Note for anyone quantizing this model themselves

gemma-4-E2B-it has tie_word_embeddings = true — there is no output.weight tensor. --output-tensor-type is therefore a silent no-op, and --token-embedding-type is the only flag that actually protects the head:

llama-quantize --output-tensor-type q6_K --token-embedding-type q6_K \
  gemma-4-E2B-it-BF16.gguf out.gguf Q4_0_ROCMFP4_COHERENT 16

Confirm it landed by reading the finished file's token_embd.weight type. And when auditing, do not grep output.weight — it false-matches blk.N.attn_output.weight and will make a healthy build look damaged.

Credits

• Base model: google/gemma-4-E2B-it — Gemma Terms of Use
• Quantization stack: ROCmFPX