Gemma-4-E4B-it-ROCmFP4-GGUF

Gemma-4-E4B-it — ROCmFP4 / ROCmFPX GGUF

First ROCmFP4/ROCmFPX quantization of google/gemma-4-E4B-it. Four quants, built from the Hub BF16 GGUF (no re-conversion from safetensors), each individually verified — text and vision — on real hardware.

Built for AMD Strix Halo (gfx1151) — Ryzen AI MAX+ 395, 128 GB unified memory — using the ROCmFPX llama.cpp fork, which adds AMD-native FP4/FP8 tensor types that mainline llama.cpp does not have.

⚠️ These files need a ROCmFPX-capable llama.cpp build. They will not load in stock llama.cpp / Ollama / LM Studio — Q4_0_ROCMFP4_* and Q*_0_ROCMFPX* are not mainline types.

Variants — all four in this repo

⚠️ One sample in this table is an outlier (102: 47.21 against a ~56 baseline). The box was otherwise idle and every other sample sits inside a 1.01x band, so this is a transient, not a bimodal model. The median is reported precisely because it is robust to it, and all five raw samples are shown rather than hidden behind a summary.

mmproj-BF16.gguf (vision projector) is included — you need it for image input.

Which to pick: the 4-bit (102). It is the smallest and the fastest by a wide margin, and it answered every correctness and vision test identically to the 8-bits. Take an 8-bit only if you want maximum fidelity for its own sake.

⚠️ Why the 4-bit is 5.76 BPW, not ~4.5

E4B is a MatFormer / per-layer-embedding model: a large fraction of its parameters live in per_layer_token_embd and related tensors that are not driven down to 4 bits. So the 4-bit build is only 1.53× smaller than the 8-bit here, where a conventional dense model would be ~1.8×. This is expected for this architecture, not a defect — the quantized layers really are FP4. One tensor, per_layer_model_proj.weight stays BF16 in all four builds. 720 tensors per file.

Measured — not estimated

Hardware: AMD Ryzen AI MAX+ 395 (Strix Halo, gfx1151), 128 GB unified. Load: -ngl 999 -c 4096 -fa on -fit off. Decode = 300 tokens, warm-up discarded, median of 3.

All spreads ≤1.03×, measured with nothing else building on the box — these are clean numbers.

A note on the usual bandwidth sanity check

For a conventional dense model you can sanity-check a decode figure with t/s × file_GB against the platform's peak memory bandwidth (~256 GB/s here) — anything well under ~80% suggests something is misconfigured. That check does not apply to this model. E4B's per-layer-embedding design means not every byte of the file is read per token, so the arithmetic returns well over 100% of "peak" and would happily exceed it. Do not read those percentages as validation for MatFormer/PLE (or MoE) architectures.

Vision was tested with a four-colour quadrant image and the model had to say which colour was where — a solid-colour square would pass trivially and proves nothing. Use -fa off for image input.

⚠️ Speculative decoding (MTP): the draft head loads, but the server aborts — don't use it

Earlier revisions of this card said MTP "does not currently work" and blamed a shared-KV / model-width problem. That explanation was wrong. The draft head initialises correctly (common_speculative_state_draft_mtp: n_embd=2560, backend_sampling=1). What actually happens:

⚠️ This repo does not ship the draft head (an earlier version of this card wrongly said it did). It lives in unsloth/gemma-4-E4B-it-GGUF under MTP/.

The -fa on abort is not a speculation bug at all — it is an uncompiled flash-attention tile case in the HIP backend, and the speculative path is simply the code path that reaches it.

Every decode number above is measured without a drafter, which is the configuration we recommend for this model today.

Verification

Each artifact was checked individually, not sampled:

• Size vs --dry-run projection — constant ~15.82 MB GGUF-header delta on all four (spread 5.9 KB) (truncation check)
• token_embd audited by exact tensor name
• Loaded, 3/3 correctness, vision content test
• Decode median of 3 with the spread reported

Exact byte sizes:

5185932160  gemma-4-E4B-it-Q4_0_ROCMFP4_COHERENT.gguf
7472712576  gemma-4-E4B-it-Q6_0_ROCMFPX_AGENT.gguf
7906233216  gemma-4-E4B-it-Q8_0_ROCMFPX.gguf
7974636416  gemma-4-E4B-it-Q8_0_ROCMFPX_AGENT.gguf

⚠️ Note for anyone quantizing this model themselves

gemma-4-E4B-it has tie_word_embeddings = true — there is no output.weight tensor. --output-tensor-type is therefore a silent no-op, and --token-embedding-type is the only flag that actually protects the head:

llama-quantize --output-tensor-type q6_K --token-embedding-type q6_K \
  gemma-4-E4B-it-BF16.gguf out.gguf Q4_0_ROCMFP4_COHERENT 16

Confirm it landed by reading the finished file's token_embd.weight type. And when auditing, do not grep output.weight — it false-matches blk.N.attn_output.weight and will make a healthy build look damaged.

Credits

• Base model: google/gemma-4-E4B-it — Gemma Terms of Use
• Quantization stack: ROCmFPX