Granite-4.0-H-Small-Cerebellum-GGUF

<p align="center"> <img src="cerebellum_banner.png" alt="Cerebellum" width="640"> </p>

Granite 4.0-H-Small — Cerebellum GGUF (14.2 GB)

Ablation-informed mixed-precision quantization of ibm-granite/granite-4.0-tiny-preview. File size: 14.2 GB. Measured WikiText-2 perplexity: 6.4580 (+1.90% vs Q3_K_M baseline of 6.3376).

This is the first Cerebellum build for a hybrid Mamba-2 + Transformer MoE architecture. The ablation revealed that routed expert weights are sensitive in this model while shared expert weights tolerate aggressive demotion — the opposite of what we expected from dense transformer MoE patterns.

Benchmarks

All benchmarks measured directly on this file.

Protocol: local llama.cpp benchmark harness on RTX 3090, temperature 0. The model-index metadata in this card's frontmatter mirrors these numbers.

What Changed

Three shared expert tensor groups (120 tensors) demoted from Q3_K to Q2_K:

What We Tested (and Kept at Q3_K)

Key Finding: Routed Experts Are Sensitive

In dense MoE models like Qwen 3.6 35B, expert gate/up/down weights tolerate Q2_K easily (+1-2%). In Granite 4.0-H-Small, the opposite is true:

• Shared experts (always active): Tolerant. Q2_K adds only +1.90% PPL.
• Routed experts (72 per layer, 10 active): Sensitive. Q2_K adds +5-13% PPL.

This is likely because Granite's expert FFN intermediate size is only 768 (vs 1536+ in larger models). Smaller weight matrices are more sensitive to quantization noise.

Architecture

How to Run

llama-server --model Granite-4.0-H-Small-Cerebellum-Q3_K_M.gguf -ngl 99 --ctx-size 4096

Fits on a 24GB GPU with room for context.

Method: Cerebellum

Cerebellum is sensitivity-guided mixed-precision quantization. We measure the PPL impact of demoting each tensor group individually, then only demote groups that stay under a threshold. Sacred tensors (routers, norms, embeddings) are never touched.

Steps:

1. Start from high-quality imatrix Q3_K_M base
2. Group tensors by function (120 shared experts, 120 routed experts, 72 Mamba projections, etc.)
3. Demote each group to Q2_K individually and measure PPL delta
4. Only ship groups that pass the threshold (+3% max)
5. Verify combined build doesn't compound beyond acceptable range

---

Quantized by deucebucket using Cerebellum methodology.