Downloads · 30 days
94
15% of all-time downloads
MichielBuisman/granite-4.0-micro-dutch-calibrated-gguf
granite-4.0-micro-dutch-calibrated-gguf is a text generation model from MichielBuisman. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
Quantized variants of IBM Granite 4.0 Micro using a Dutch importance matrix derived from the Leesplank corpus.
Downloads · 30 days
94
15% of all-time downloads
All-time downloads
644
Public
Repo size
13 GB
Likes
2
Public
Click a slice to open those files.
.gguf13 GB · 100%
From the Hugging Face model README
Quantized variants of IBM Granite 4.0 Micro using a Dutch importance matrix derived from the Leesplank corpus.
| Model | Size | Purpose |
|---|---|---|
| granite-4.0-micro-Q4_K_NL | ~2.0 GB | Primary deployment target |
| granite-4.0-micro-Q4_K_NL_plain | ~1.9 GB | Control: imatrix only, no layer map |
| granite-4.0-micro-Q5_K_NL | ~2.4 GB | Preferred quality target |
| granite-4.0-micro-Q5_K_NL_plain | ~2.3 GB | Control: imatrix only, no layer map |
| granite-4.0-micro-Q6_K_NL | ~3.3 GB | Quality ceiling reference |
FP16
↓
Dutch imatrix calibration
↓
Layer promotion (Unsloth map)
↓
Quantization (4, 5, 6)
↓
Evaluation (PPL + KLD)
Evaluation uses the same stratified cluster sampling used to construct the calibration imatrix, but with held-out texts not used during calibration. Evaluation uses token-level perplexity averaged per text, then aggregated by cluster.
Dutch-calibrated Q4_K_M quantization of IBM Granite 4.0 Micro Dense (3B), using a 295K-token Dutch imatrix and Unsloth's per-tensor layer promotion map extracted from their Q4_K_XL GGUF. Produces lower perplexity on Dutch text than standard Q4_K_M while fitting in ~2GB VRAM.
This is the primary deployment target in the Dutch Granite quantization series. See the dataset card for full methodology.
Standard Q4_K_M uses a generic imatrix (or none at all) and applies uniform quantization across all tensor types. This quantization improves on that in two ways:
Dutch imatrix: The importance matrix was built from 3,504 Dutch texts (~295K tokens) drawn from a stratified sample of the Leesplank corpus — 70 semantic clusters × 6 complexity strata (simple/complex × low/mid/high Levenshtein distance). This tells the quantizer which weights matter most for Dutch language modeling specifically, not English or generic multilingual text.
Unsloth layer map: Rather than promoting layers heuristically, the per-tensor quantization types were extracted directly from Unsloth's published granite-4.0-micro-UD-Q4_K_XL GGUF. Unsloth's XL series applies careful per-tensor analysis to identify which layers are sensitive. Extracting that map and applying it with a Dutch imatrix combines their structural insight with Dutch-specific calibration.
Promoted tensors (above Q4_K_M base):
attn_norm and ffn_norm weights → F32attn_v weights → Q6_Kffn_down weights → Q6_K (layers 0,1,2,3,4,6,7,16,19,22,25,28,31,34,35,36,37,38,39)token_embd → Q6_Koutput_norm → F32| Property | Value |
|---|---|
| Base model | ibm-granite/granite-4.0-micro |
| Architecture | Granite 4.0 Dense (40 layers, 3B parameters) |
| Quantization | Q4_K_M with Dutch imatrix + Unsloth layer map |
| Calibration tokens | ~295K Dutch (Leesplank corpus) |
| Calibration clusters | 70 semantic clusters, 6 strata each |
| File size | ~2.0 GB |
| Context length | 128K (as per base model) |
| Chat template | Granite instruct |
Measured on 66 Dutch text clusters (12,600 texts total, same stratified sample — see dataset card). FP16 baseline median PPL: 8.938.
| Model | Median PPL | Mean Δ vs FP16 | Mean % Δ |
|---|---|---|---|
| FP16 baseline | 8.938 | — | — |
| Q4_K_NL (this model) | 9.063 | +0.131 | +1.50% |
| Q4_K_NL_plain (imatrix only) | 9.099 | +0.170 | +1.96% |
| Q4_K_XL (Unsloth reference) | 9.054 | +0.122 | +1.39% |
| IBM Q4_K_M (plain, no imatrix) | 9.458 | +0.480 | +5.47% |
The Dutch imatrix + Unsloth layer map narrows the PPL gap relative to vanilla Q4_K_M while landing very close to Unsloth's Q4_K_XL reference (0.131 vs 0.122 — note Q4_K_NL is slightly behind Unsloth Q4 in mean delta, but beats it in median). The plain imatrix-only variant sits 0.039 PPL points above this model, confirming the layer map contributes meaningfully on top of calibration alone.
Note: If VRAM allows, Q5_K_NL achieves dramatically better fidelity (within 0.064% of FP16 on average). Use Q4_K_NL when the 2GB target matters.
KLD measures how closely the full output probability distribution matches FP16 at each token position — more sensitive than perplexity, which only captures the most likely token. Measured across the same 70 clusters.
| Model | Weighted mean KLD | Weighted median KLD |
|---|---|---|
| Q4_K_NL (this model) | 12.918 | 11.604 |
| Q4_K_XL (Unsloth) | 12.933 | 11.695 |
| IBM Q4_K_M | 12.943 | 11.626 |
| Q5_K_NL | 12.955 | 11.691 |
| Q6_K_XL (Unsloth) | 12.972 | 11.690 |
| Q6_K_NL | 12.993 | 11.661 |
| Q5_K_XL (Unsloth) | 13.000 | 11.792 |
Total spread across all 7 models: 0.082 nats. This confirms granite-4.0-micro quantizes robustly regardless of calibration or bit depth — all variants preserve the full distribution similarly well. Q4_K_NL edges the ranking but the margin is within noise.
# LM Studio: search for this model name, download, use Granite chat template
# llama.cpp:
llama-cli -m granite-4.0-micro-Q4_K_NL.gguf -ngl 99 -p "Vertel me over ..."
The layer map extraction script (05b_extract_unsloth_layers.py) and quantization script (06_quantize.py) are available at: https://github.com/okeribok/dutchdynamicquant
If you use this model, please cite the dataset card and IBM's original Granite 4.0 release.
Dutch-calibrated Q4_K_M quantization of IBM Granite 4.0 Micro Dense (3B), using a 295K-token Dutch imatrix without per-tensor layer promotion. Control variant for comparing the contribution of the Unsloth layer map vs calibration alone.
This model uses the same Dutch imatrix as Q4_K_NL but applies standard Q4_K_M quantization uniformly — no tensor-type overrides. It exists to isolate the contribution of the Unsloth layer map. See the dataset card for full methodology.
| Property | Value |
|---|---|
| Base model | ibm-granite/granite-4.0-micro |
| Architecture | Granite 4.0 Dense (40 layers, 3B parameters) |
| Quantization | Q4_K_M with Dutch imatrix, no layer overrides |
| Calibration tokens | ~295K Dutch (Leesplank corpus) |
| File size | ~1.9 GB |
| Context length | 128K |
| Chat template | Granite instruct |
Use Q4_K_NL (the dynamic variant) for deployment — it has the layer map applied and shows lower perplexity. Use this model only if you want to verify the layer map contribution for your specific use case, or if you find Q4_K_NL behaves unexpectedly.
| Model | Median PPL | Mean Δ vs FP16 | Mean % Δ |
|---|---|---|---|
| FP16 baseline | 8.938 | — | — |
| Q4_K_NL (with layer map) | 9.063 | +0.131 | +1.50% |
| Q4_K_NL_plain (this model) | 9.099 | +0.170 | +1.96% |
The layer map reduces the PPL gap by ~23% relative (0.170 → 0.131). Both models beat a vanilla IBM Q4_K_M by a wide margin (+5.47% without any imatrix or layer map).
Dutch-calibrated Q5_K_M quantization of IBM Granite 4.0 Micro Dense (3B), using a 295K-token Dutch imatrix and Unsloth's per-tensor layer promotion map extracted from their Q5_K_XL GGUF. Recommended if VRAM allows — achieves perplexity within 0.064% of FP16 on average on Dutch text.
This is the preferred quality target in the Dutch Granite quantization series. The Q5 Unsloth layer map promotes significantly more tensors than Q4 — adding Q, K, and gate/up weights for early and late layers (0,1,2,6,39) on top of the V and ffn_down promotions. See the dataset card for full methodology.
| Property | Value |
|---|---|
| Base model | ibm-granite/granite-4.0-micro |
| Architecture | Granite 4.0 Dense (40 layers, 3B parameters) |
| Quantization | Q5_K_M with Dutch imatrix + Unsloth layer map |
| Calibration tokens | ~295K Dutch (Leesplank corpus) |
| File size | ~2.4 GB |
| Context length | 128K |
| Chat template | Granite instruct |
On top of Q5_K_M base, these tensors are elevated:
attn_norm and ffn_norm → F32attn_v → Q6_K (all 40 layers)attn_q and attn_k → Q6_K for layers 0,1,2,6,39ffn_gate and ffn_up → Q6_K for layers 0,1,2,6,39ffn_down → Q6_K (17 layers) or Q8_0 (layers 6,39)token_embd → Q6_K, output_norm → F32Layers 0-2 (early), 6 (attention transition), and 39 (final layer) receive the most promotion — consistent with the known sensitivity of first and last transformer blocks.
Measured on 66 Dutch text clusters (12,600 texts). FP16 baseline median PPL: 8.938.
| Model | Median PPL | Mean Δ vs FP16 | Mean % Δ | Clusters beating FP16 |
|---|---|---|---|---|
| FP16 baseline | 8.938 | — | — | — |
| Q5_K_NL (this model) | 8.952 | +0.006 | +0.064% | 24 / 66 |
| Q5_K_NL_plain (imatrix only) | 9.027 | +0.064 | +0.707% | 0 / 66 |
| Q5_K_XL (Unsloth reference) | 8.981 | +0.036 | +0.423% | 6 / 66 |
| Q6_K_NL | 8.962 | +0.025 | +0.282% | 1 / 66 |
Q5_K_NL achieves a mean PPL delta of only +0.006 vs FP16 — and actually beats FP16's measured perplexity on 36% of clusters. This is not noise: the Dutch calibration data is guiding the quantizer well enough that rounding errors partially cancel across Dutch text sequences. Unsloth's own Q5 sits at +0.036, nearly 6× larger a gap. The plain imatrix-only variant at +0.064 shows the layer map contributes roughly a 10× improvement at Q5 (from 0.064 to 0.006) — a much larger gain than at Q4.
All Dutch Granite quants show very similar full-distribution fidelity to FP16. The total spread across all 7 tested models is only 0.082 nats. Q5_K_NL sits mid-pack in KLD (12.955 weighted mean vs Q4_K_NL's 12.918 best), which is consistent with the PPL picture: Q5 preserves the most likely token exceptionally well, while Q4 has a marginal advantage in preserving the tail of the distribution. In practice, neither difference is perceptible.
# LM Studio: search for this model name, download, use Granite chat template
# llama.cpp:
llama-cli -m granite-4.0-micro-Q5_K_NL.gguf -ngl 99 -p "Vertel me over ..."
The layer map extraction script (05b_extract_unsloth_layers.py) and quantization script (06_quantize.py) are available at: https://github.com/okeribok/dutchdynamicquant
If you use this model, please cite the dataset card and IBM's original Granite 4.0 release.
Dutch-calibrated Q5_K_M quantization of IBM Granite 4.0 Micro Dense (3B), using a 295K-token Dutch imatrix without per-tensor layer promotion. Control variant for comparing Q5_K_NL vs calibration-only Q5.
Same Dutch imatrix as Q5_K_NL, standard Q5_K_M quantization with no tensor-type overrides. Exists to isolate the Unsloth layer map contribution at Q5 bit width. See the dataset card for full methodology.
| Property | Value |
|---|---|
| Base model | ibm-granite/granite-4.0-micro |
| Architecture | Granite 4.0 Dense (40 layers, 3B parameters) |
| Quantization | Q5_K_M with Dutch imatrix, no layer overrides |
| Calibration tokens | ~295K Dutch (Leesplank corpus) |
| File size | ~2.3 GB |
| Context length | 128K |
| Chat template | Granite instruct |
| Model | Median PPL | Mean Δ vs FP16 | Mean % Δ |
|---|---|---|---|
| FP16 baseline | 8.938 | — | — |
| Q5_K_NL (with layer map) | 8.952 | +0.006 | +0.064% |
| Q5_K_NL_plain (this model) | 9.027 | +0.064 | +0.707% |
At Q5, the Unsloth layer map delivers a 10× reduction in mean PPL gap (0.064 → 0.006). Use Q5_K_NL for deployment.
Dutch-calibrated Q6_K quantization of IBM Granite 4.0 Micro Dense (3B) with Unsloth's Q6 layer promotion map. Near-FP16 quality ceiling; use this as a reference when measuring quantization degradation, or when quality is critical and VRAM allows.
The Q6 Unsloth layer map is the most aggressive — it promotes all Q, K, V attention weights across all 40 layers to Q8_0, along with selected ffn weights, leaving almost nothing at base Q6_K except the least sensitive feed-forward mid-layers. Combined with the Dutch imatrix this produces a GGUF very close to FP16 quality on Dutch text. See the dataset card for full methodology.
| Property | Value |
|---|---|
| Base model | ibm-granite/granite-4.0-micro |
| Architecture | Granite 4.0 Dense (40 layers, 3B parameters) |
| Quantization | Q6_K with Dutch imatrix + Unsloth layer map |
| Calibration tokens | ~295K Dutch (Leesplank corpus) |
| File size | ~3.3 GB |
| Context length | 128K |
| Chat template | Granite instruct |
On top of Q6_K base, these tensors are elevated to Q8_0:
attn_q, attn_k, attn_v (all 40 layers) → Q8_0ffn_down, ffn_gate, ffn_up for layers 0,1,2,6,9,39 → Q8_0token_embd → Q8_0, output_norm → F32, all norm layers → F32This is effectively a hybrid Q6/Q8 model with F32 norms throughout.
Measured on 66 Dutch text clusters (12,600 texts). FP16 baseline median PPL: 8.938.
| Model | Median PPL | Mean Δ vs FP16 | Mean % Δ |
|---|---|---|---|
| FP16 baseline | 8.938 | — | — |
| Q6_K_NL (this model) | 8.962 | +0.025 | +0.282% |
| Q6_K_XL (Unsloth reference) | 8.971 | +0.027 | +0.303% |
| Q5_K_NL | 8.952 | +0.006 | +0.064% |
Notably, Q5_K_NL achieves better average perplexity than Q6_K_NL (+0.064% vs +0.282%). This is not a bug: at Q6 the Dutch imatrix and layer map have less room to improve on the base, and the marginal benefits of Dutch calibration are smaller. Q5_K_NL is the sweet spot for Dutch text fidelity in this series.
| Model | Weighted mean KLD | Weighted median KLD |
|---|---|---|
| Q4_K_NL | 12.918 | 11.604 |
| Q4_K_XL (Unsloth) | 12.933 | 11.695 |
| IBM Q4_K_M | 12.943 | 11.626 |
| Q5_K_NL | 12.955 | 11.691 |
| Q6_K_XL (Unsloth) | 12.972 | 11.690 |
| Q6_K_NL (this model) | 12.993 | 11.661 |
| Q5_K_XL (Unsloth) | 13.000 | 11.792 |
Total spread: 0.082 nats across all 7 models. All variants preserve the full output distribution to essentially the same degree — granite-4.0-micro is a very quantization-stable model. Q6_K_NL's slightly higher KLD than Q4_K_NL is a second-decimal artefact without practical significance.
| Model | Size | Median PPL | Mean Δ FP16 | Purpose |
|---|---|---|---|---|
| granite-4.0-micro-Q4_K_NL | ~2.0 GB | 9.063 | +1.50% | Primary deployment target |
| granite-4.0-micro-Q4_K_NL_plain | ~1.9 GB | 9.099 | +1.96% | Control: imatrix only, no layer map |
| granite-4.0-micro-Q5_K_NL | ~2.4 GB | 8.952 | +0.064% | Recommended quality target |
| granite-4.0-micro-Q5_K_NL_plain | ~2.3 GB | 9.027 | +0.707% | Control: imatrix only, no layer map |
| granite-4.0-micro-Q6_K_NL | ~3.3 GB | 8.962 | +0.282% | Quality ceiling reference |