Granite-4.1-30B-ROCmFP4-GGUF

Granite-4.1-30B — ROCmFP4 / ROCmFPX GGUF

First ROCmFP4/ROCmFPX quantization of ibm-granite/granite-4.1-30b. Four quants, all built from IBM's own BF16 GGUF (no re-conversion from safetensors), all verified on real hardware.

Built for AMD Strix Halo (gfx1151) — Ryzen AI MAX+ 395, 128 GB unified memory, ROCm 7.2.4 — using the ROCmFPX llama.cpp fork, which adds AMD-native FP4/FP8 tensor types that mainline llama.cpp does not have.

⚠️ These files require a ROCmFPX-capable llama.cpp build. They will not load in stock llama.cpp / Ollama / LM Studio — the Q4_0_ROCMFP4_* and Q*_0_ROCMFPX* tensor types are not in mainline.

Variants — all four in this repo

Which to pick: the 4-bit (102) unless you specifically need 8-bit fidelity — it is 1.85× faster and less than half the size. The AGENT variants keep more attention-output tensors at higher precision, which helps tool-calling and (on models that have one) speculative-draft acceptance; Granite-4.1-30b ships no draft head, so here AGENT costs a little speed for a little precision.

⚠️ Q6_0_ROCMFPX_AGENT is not a middle rung — it lands at 7.43 BPW and performs like the 8-bits. Choose it for precision, not for size.

Measured — not estimated

Hardware: AMD Ryzen AI MAX+ 395 (Strix Halo, gfx1151), 128 GB unified, ROCm 7.2.4. llama-bench -n 300 -p 512 -r 3, warm-up discarded, idle box. Median of 3; all three runs shown.

These are the deployed numbers. Granite-4.1-30b ships no MTP/EAGLE/draft head, so there is no speculative-decoding multiplier being left on the table.

Why you can trust the speed figures

Weights stream once per token, so t/s × file_GB gives effective memory bandwidth. Against Strix Halo's ~256 GB/s peak:

All four sit at 81–83% of peak, and the 4-bit/8-bit speed ratio (1.85×) tracks the inverse size ratio (1.82×). That is the signature of a purely bandwidth-bound dense model running at the hardware ceiling — nothing is misconfigured or spilling to CPU.

Verification

Every artifact was checked individually:

• Loaded with -ngl 999 -c 4096 -fa on -fit off
• Correctness 3/3 at max_tokens: 1024 — 17*23 → 391, capital of Japan → Tokyo, days in 2024 → 366
• Size vs --dry-run projection — each file exceeds its projection by a constant GGUF-header delta (+3,576,564 B; +3,579,186 B for the 4-bit), which is the truncation check
• Head protection audited by exact tensor name, not by trusting the flag (see below)

Exact byte sizes:

16348341472  granite-4.1-30b-Q4_0_ROCMFP4_COHERENT.gguf
26820404448  granite-4.1-30b-Q6_0_ROCMFPX_AGENT.gguf
29785777376  granite-4.1-30b-Q8_0_ROCMFPX.gguf
30272316640  granite-4.1-30b-Q8_0_ROCMFPX_AGENT.gguf

⚠️ Note for anyone quantizing this model themselves

granite-4.1-30b has tie_word_embeddings = true — there is no output.weight tensor at all. That means --output-tensor-type is a silent no-op on this model, and --token-embedding-type is the only flag that actually protects the head:

llama-quantize --output-tensor-type q6_K --token-embedding-type q6_K \
  granite-4.1-30b-bf16-00001-of-00005.gguf out.gguf Q4_0_ROCMFP4_COHERENT 16

Verify it landed by reading the finished file's token_embd.weight type — do not assume it worked because the flag was accepted. Also: do not grep output.weight when auditing, because it false-matches blk.N.attn_output.weight and will make a healthy build look damaged.

Model

Dense, 64 layers, hidden 4096, FFN 32768, GQA 32/8 heads, vocab 100352, context 131072. Source: ibm-granite/granite-4.1-30b-GGUF BF16 shards (00001-of-00005), quantized directly — no safetensors re-conversion.

Credits

• Base model: IBM Granite (Apache 2.0)
• Quantization stack: ROCmFPX