Mellum2-12B-A2.5B-Thinking-APEX-GGUF

Mellum2-12B-A2.5B-Thinking — APEX GGUF

Measured-allocation GGUF quants of JetBrains/Mellum2-12B-A2.5B-Thinking (12B total / 2.5B active MoE, 28 layers, 64 experts, thinking, Apache-2.0).

Calibrated with a code-weighted imatrix (~70% multi-language code, 336 chunks) and evaluated by KL divergence against the bf16 model, not perplexity alone.

Why this model needs deliberate allocation

Mellum2's moe_intermediate_size is 896, and 896 is not divisible by 256. Every K-quant type (Q6_K/Q5_K/Q4_K/Q3_K/IQ4_XS/IQ3_/IQ2_) requires rows divisible by 256, so ffn_down_exps — 3.70B params, 30.4% of the model — cannot take a K-quant at all. Everything else is fine (hidden 2304, FFN 7168, vocab 98304).

Stock Q4_K_M handles this by falling back per tensor. Its mix wants Q6_K on every other layer's ffn_down; unable to, it falls upward to Q8_0:

So a third of the model receives bits by fallback rather than by choice, while the K-quantable 70% sits at Q4_K. These quants set ffn_down_exps explicitly instead.

Tiers

Measured quality — KL divergence vs bf16

Paired against bf16 logits over 100 chunks of a code-heavy corpus (no wikitext contamination).

compact is 2.4× closer to bf16 than stock Q4_K_M for 0.22 bpw more (0.33 GB) — mean KLD 0.0264 vs 0.0633, and +1.54 points of top-token agreement.

mini is the honest trade in the other direction: 14% smaller than stock Q4_K_M and measurably further from bf16. It is the smallest sensible tier for this model — below IQ4_NL the only legal step for that 30% is Q2_0 at 2.25 bpw, so there is a cliff rather than a slope.

Choosing

• compact — the recommended default. Best fidelity per byte here by a wide margin.
• i-quality — when you want the closest match to bf16.
• mini — when 6.92 GB is the constraint and you accept the quality cost.

Notes

• Perplexity was too blunt to separate these: every difference sat inside its ±0.019 error bars, and on one comparison PPL ranked the models in the opposite order to KL divergence. PPL scores only the observed token's probability; KLD scores the whole distribution. The KLD numbers above are the ones to trust.
• Requires a llama.cpp build with LLM_ARCH_MELLUM support.