Mellum2-12B-A2.5B-Instruct-ROCmFP4-GGUF

🔧 Runtime: build the ROCmFPX fork below

Stock llama.cpp will not load this file. You need both the mellum architecture and the ROCmFP4 tensor types in one tree. Upstream charlie12345/ROCmFPX has the ROCmFP4 types but not mellum. Our fork has both:

kingjones30/ROCmFPX — a fork of charlie12345/ROCmFPX, branch main.

git clone https://github.com/kingjones30/ROCmFPX.git
cd ROCmFPX
cmake -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-server llama-quantize -j$(nproc)

Verified 2026-08-27 on gfx1151: clean clone → 0 build errors → llama-server loads a mellum ROCmFP4 GGUF from this family and generates coherent text.

⚠️ STOCK llama.cpp WILL NOT LOAD THIS MODEL

The Mellum architecture is not merged upstream. Ignore the auto-generated "Use this model" commands above — build the ROCmFPX fork linked just below.

🚀 96.92 tok/s on AMD Ryzen AI MAX+ 395 (gfx1151 / Strix Halo) — 6.49 GiB, 1.12 GiB smaller and 7.2% faster than Q4_K_M.

✅ The patch you need is in this repo

patches/rocmfpx-2809dc5-add-bailingmoe3qlora-mellum-zaya.patch — applies to charlie12345/ROCmFPX at commit 2809dc5, verified with git apply --check.

git clone https://github.com/charlie12345/ROCmFPX.git && cd ROCmFPX
git checkout 2809dc5
git apply patches/rocmfpx-2809dc5-add-bailingmoe3qlora-mellum-zaya.patch
cmake -B build -S . -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc)

Full build notes, per-architecture details and licence: patches/README.md in this repo.

⚠️ If you add files under src/models/, re-run cmake -B build -S . — the models/*.cpp GLOB is configure-time, so cmake --build alone will not link them.

---

Mellum2-12B-A2.5B-Instruct — ROCmFP4 (tier 102 COHERENT) GGUF

A 4-bit ROCmFP4 quantization of JetBrains/Mellum2-12B-A2.5B-Instruct, built for AMD gfx1151 (Ryzen AI MAX+ 395 / Strix Halo), with the LM head and token embeddings held at Q6_K.

Smaller and faster than Q4_K_M on the target hardware — see below.

---

⛔ REQUIRES A PATCHED llama.cpp — STOCK WILL NOT LOAD THIS

mellum is not in mainline llama.cpp. Support is open in PR #23966 ("model: add Mellum architecture", Xarbirus; branch Xarbirus/llama.cpp:mellum2), unmerged at time of writing. The ROCmFP4 quant types additionally require a fork that implements them — upstream has no Q4_0_ROCMFP4_*.

⚠️ strings is not a capability check

Our build's libllama.so contained the literal string mellum and still failed with unknown model architecture: 'mellum'. The string lives in a name table; the loader is separate code. Grepping the binary tells you nothing — attempt the load.

---

All quant variants

All measured on one box, one binary (Ryzen AI MAX+ 395, gfx1151, ROCm 7.2.4), median of 3, warm-up discarded — so these rows are directly comparable.

Repos: 4-bit · 8-bit AGENT · 8-bit plain

AGENT is faster here — 74.93 vs 72.76, ranges disjoint (+3.0%). Both 8-bit builds are well below the 4-bit build's 104.99 tok/s; they exist for accuracy headroom, not speed.

On AGENT generally: it keeps more tensors at true Q8_0 instead of the packed 8-bit type. That raises MTP draft acceptance on models which have an MTP head (measured +6.2% on Qwen3.8-27B). Mellum2 has no MTP head, so there is nothing for the extra precision to feed and the two 8-bit builds differ only marginally — in either direction.

Measured results

Ryzen AI MAX+ 395 (gfx1151, 128 GB unified, ROCm 7.2.4), -ngl 99 -c 4096 -fa on.

+7.2% decode over Q4_K_M while 1.12 GiB smaller.

Per-tensor types (audited in the finished file, 339 tensors)

tie_word_embeddings is false on this model, so a real output.weight exists and both --output-tensor-type and --token-embedding-type apply. (On a tied model --output-tensor-type is a silent no-op — worth checking before you trust it.)

Mellum2 has no shared experts and no SSM/conv state, so the protections that matter for hybrid architectures do not apply here. Its layer_types alternate sliding_attention ×3 → full_attention (n_swa = 1024), and the loader honours that pattern per layer.

---

What was NOT measured

• No perplexity run, and no quality A/B against Q4_K_M or BF16. The checks above are memorized-fact prompts — necessary but not sufficient; a damaged model can pass them.
• No code-generation benchmark. This is a coding model and we did not evaluate it as one.
• No long-context testing (the model supports 131,072; nothing was run near it).
• No tool-calling evaluation.
• Speed figures are single measurements per build on one machine, not medians of repeated runs.

---

Model

MellumForCausalLM / mellum. 28 layers · hidden 2304 · vocab 98,304 · 64 experts, 8 active · moe_intermediate_size 896 · sliding/full attention interval 4 · context 131,072 · tie_word_embeddings: false.

Base model licence: Apache-2.0 (inherited). All credit for the model itself goes to JetBrains.