granite-4.0-h-small-APEX-GGUF

Granite-4.0-H-Small — APEX GGUF

MoE-aware, mixed-precision APEX quantizations of ibm-granite/granite-4.0-h-small — IBM's hybrid Mamba-2 / Transformer MoE (granitemoehybrid): 40 layers (36 Mamba-2 + 4 attention), 64-routed + shared-expert MoE, ~32B total / ~9B active, Apache-2.0. To my knowledge the first APEX quant of a 32B Granite hybrid.

Results & which file to pick

Perplexity on wikitext-2-raw (test, 200×512-token windows), llama-perplexity. All quants are imatrix-guided (diverse calibration).

• i-quality — recommended; near-lossless (+0.5%). Q6_K/Q5_K experts, Q8_0 shared, Q6_K attn/SSM.
• i-compact — Q3_K mid experts; a real but usable quality step (+4.3%) for tighter fits.
• i-mini — IQ2_S mid experts; smallest that stays coherent (16 GB cards), but a clear quality drop (+12.5%) — pick it only when you must fit and can tolerate the hit.

Note on minimum size: Granite's routed-expert dim is 640 (not 256-divisible), so i-quants can't apply to the ffn_down_exps — they fall back to ~Q4_K. That floors the practical minimum around ~13 GB (i-mini) and, as a bonus, protects the down projections' quality. Going below that (nano/IQ1) buys little here.

Usage (llama.cpp)

llama-cli   -m granite-4.0-h-small-APEX-i-quality.gguf -ngl 999 -p "Hello"
llama-server -m granite-4.0-h-small-APEX-i-compact.gguf -ngl 999 --host 0.0.0.0 --port 8080

Requires a llama.cpp build supporting the granitemoehybrid architecture.

Method

APEX = per-tensor-role, per-layer precision over stock llama-quantize --tensor-type-file. Granite-H needed its Mamba-2 mixer tensors added to the map (the stock generator omits them):

• Mamba-2 (36 layers): ssm_in, ssm_conv1d, ssm_out at mixer precision; 1-D state (ssm_a/d/dt/norm) left F32.
• Attention (4 layers): attn_q/k/v/output.
• MoE (all 40): routed ffn_*_exps on a depth gradient; shared ffn_*_shexp at Q8_0/Q6_K.

Baseline: IBM's own f16 GGUF. imatrix from Bartowski calibration_datav3. Config gen + patcher: configs/, patch_granite_config.py, REPRODUCE.md.

Null result: protecting the Mamba-2 recurrence doesn't help

A hand-roll pinning ssm_in/out/conv1d to Q8_0 (22 GB, PPL 5.779) was within noise of i-quality — the SSM/recurrence tensors aren't precision-sensitive. Consistent with the same finding on Kimi-Linear's KDA and Granite-4.0-H-Tiny.

It is shipped here as granite-4.0-h-small-APEX-handroll.gguf for transparency and reproduction, but there's no reason to use it — it's slightly larger than i-quality (22 GB vs 21 GB) for no measurable quality gain (PPL 5.779 vs 5.781). Use i-quality.

Attribution & licenses

See LICENSE (Apache-2.0) and NOTICE.

• Base: IBM (@ibm-granite) — granite-4.0-h-small (Apache-2.0)
• Engine: llama.cpp (@ggml-org) (MIT)
• APEX: Ettore Di Giacinto / LocalAI (@mudler) — localai-org/apex-quant (MIT)
• Calibration: Bartowski (@bartowski) — calibration_datav3

Unofficial community quantization; not affiliated with or endorsed by IBM.