granite-4.2-3b-Q4_K_M-imatrix-GGUF

Granite-4.2-3B — Q4_K_M GGUF (imatrix)

A 4-bit GGUF of ibm-granite/granite-4.2-3b, quantized with an importance matrix built from a mixed prose and tool-calling calibration corpus. Of the three quantizations we made of this model, it's the closest to the original and the smallest.

"Same top token" is llama.cpp's Same top p statistic: the share of positions where the quantized model's most likely next token matches the f16 model's.

How it was made

1. Convert: convert_hf_to_gguf.py (mainline llama.cpp), bf16 safetensors → f16 GGUF. Granite's scaling parameters (attention_scale, embedding_scale, residual_scale, logits_scale) are supported natively.
2. Importance matrix: llama-imatrix over bartowski's v6 calibration dataset: prose plus tool-calling conversations, rendered through Granite's own chat template. 573 chunks of 512 tokens.
3. Quantize: llama-quantize --imatrix <imatrix> <f16.gguf> <out.gguf> Q4_K_M.
4. Evaluate: llama-perplexity --kl-divergence against the f16 model's saved logits, on the wikitext-2-raw test split.

The calibration corpus matters. This build beat our best GPTQ build (0.110 KLD) by about 44%. Plain wikitext-2 is a common default for imatrix calibration, but a corpus in the model's own chat format with tool use covers more of what the model actually does.

Usage

llama-server -m granite-4.2-3b-Q4_K_M.gguf

Any llama.cpp-based runtime that supports Granite 4 should load it. We haven't benchmarked this file's inference speed yet. The speed figures in the GPTQ card are for vLLM and don't apply here.

Which one should I use?

• llama.cpp, Ollama, LM Studio, or similar: this GGUF. It's the best quality and the smallest of the three.
• vLLM: the GPTQ W4A16 build, which has measured serving numbers on RDNA2.
• MXFP4 research: the MXFP4 build (compressed-tensors and GGUF). It's an experiment, not the recommended pick.

All quantizations compared

Lower KLD means the quantized model's next-token distribution stays closer to the original's.

How to read this table. The GGUF rows come from a different harness than the others.

• GGUF rows: llama-perplexity --kl-divergence against the f16 GGUF's logits, on the wikitext-2-raw test split.
• GPTQ and MXFP4 (compressed-tensors): a transformers script that loads each model next to the bf16 original and compares them over 51,100 tokens of the same wikitext-2 test split.

Both measure mean KL divergence on the same corpus, but the code isn't identical. Compare rows within the same harness directly; across harnesses, treat gaps as directional, not decimal-precise. The MXFP4 weights were measured both ways (0.1405 in transformers, 0.138 in llama.cpp), which gives a feel for how closely the two harnesses agree.

Hardware: all measurements are on an AMD Radeon Pro V620 (RDNA2, gfx1030, 32 GB) with ROCm 7.14. Nothing here was tested on NVIDIA or other AMD GPUs.

Credits

• Base model: ibm-granite/granite-4.2-3b by IBM, Apache 2.0. All credit for the model itself goes to IBM's Granite team. These are unofficial quantizations, not affiliated with or endorsed by IBM.
• Calibration data: bartowski's imatrix calibration dataset (v6) and its render_corpus.py.
• Tools: llama.cpp.