Downloads · 30 days
174
20% of all-time downloads
darioooooo0o/granite-4.2-8b-G-Q5-GGUF
granite-4.2-8b-G-Q5-GGUF is a machine learning model from darioooooo0o. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for gguf. The card lists the license as apache-2.0.
Downloads · 30 days
174
20% of all-time downloads
All-time downloads
854
Public
Repo size
10.5 GB
Likes
1
Public
Click a slice to open those files.
.gguf5.6 GB · 100%
From the Hugging Face model README
Requests, questions or suggestions? Message me on X: https://x.com/imdariotoo
Two deliverables here:
| Tensor | Type |
|---|---|
| attn_q / attn_k / attn_output | Q5_K |
| attn_v | Q6_K |
| ffn_down | Q5_K |
| ffn_gate / ffn_up | Q4_K |
| token_embd | Q5_K |
| output | Q6_K |
Fidelity = KL divergence vs bf16 over 30x512-token chunks of held-out wiki text.
| Quant | Size GiB | KLD ↓ | Same-top-p ↑ | tg128 | pp512 |
|---|---|---|---|---|---|
| Q4_K_M (official IBM) | 5.10 | 0.1796 | 86.0% | 55.6 | 1760 |
| Q4_K_M (bartowski) | 5.16 | 0.0996 | 90.4% | 53.5 | 1739 |
| IQ4_XS (bartowski) | 4.60 | 0.1149 | 87.9% | 59.9 | 1918 |
| Q4_K_L (bartowski) | 5.44 | 0.0911 | 90.4% | 53.0 | 1738 |
| G-Q5 (this repo) | 5.22 | 0.0809 | 91.2% | 50.5-53.8 | ~1700 |
Also tested and rejected:
The official Q4_K_M's KLD is nearly 2x worse than bartowski's free imatrix upgrade of the same quant. If you use granite from the official GGUF repo, you're leaving real accuracy on the table.
| KV dtype | Max context |
|---|---|
| q4_0/q4_0 | 108K verified (109K+ OOM) |
| q8_0/q8_0 | <96K |
| f16/f16 | OOM past ~30K |
KV cost is architecture-determined: 40 full-attention layers x 8 KV heads x 128 head_dim = ~80KB/token at f16, ~22KB at q4_0. Hybrid-attention models (qwen3.5-style: SSM layers between attention) need 1/4 the KV per token — no weight quantization can close that gap.
At 2.3K-token prompts: q4_0 KV = q8_0 KV within noise (pp ~1600, tg ~46 both). KV dtype only matters at deep fill (50K+). Use q8_0 KV for free quality; drop to q4_0 only to extend max context.
During a real agent session (hermes-core suite), median decode was 29 tok/s vs 50+ on clean llama-bench runs. Causes: growing context per task (attention cost scales with KV depth — long generations ran at 26 vs 32 for short ones) and per-request churn (127 requests in one suite). Plan capacity on the agent number, not the bench number.
| Method | Result |
|---|---|
| ngram-mod self-speculative | +31% decode on repetitive/structured workloads, ~neutral on open-ended |
| granite-4.2-3b as classic draft | Net loss (-43%): drafter is also 40 layers, its pass costs more than it saves |
Same tokenizer != fast draft. Draft speedup requires draft model depth << target depth or a trained single-pass head (DFlash/EAGLE style).
We tested greedy (temp=0) vs IBM's official config (temp=1.0, top_p=0.95) across 10 varied prompts: 0/10 spirals for greedy vs 1/10 for official — no significant difference. An earlier single-run observation that temp 0 "causes reasoning loops" did not replicate; the loop we saw was on a heavily-quantized experimental variant, not this one. Use IBM's official config (what the model was tuned with), and give thinking mode a large budget (8K+) for math/code tasks.
llama-quantize \
--imatrix imatrix-granite42-mixed.gguf \
--tensor-type-file tensor_types_G.txt \
granite-4.2-8b-bf16.gguf granite-4.2-8b-G-Q5.gguf Q4_K_S 32
<details><summary>tensor_types_G.txt</summary>
attn_q=q5_k
attn_k=q5_k
attn_v=q6_k
attn_output=q5_k
ffn_down=q5_k
ffn_gate=q4_k
ffn_up=q4_k
token_embd.weight=q5_k
output.weight=q6_k
</details>
granite-4.2-8b-G-Q5-Q5_K_M.gguf — the quant (dominant type Q5_K; mixed per-tensor recipe above)imatrix-granite42-mixed.gguf — importance matrix used (reusable)calibration_mixed.txt — calibration corpustensor_types_G.txt — per-tensor quantization mapRan the hermes-core tool-use suite (13 tasks, sequential, real Hermes agent + tool loop) on G-Q5.
Initial run: 6/13 (46%) — but 5 of the 7 failures were infrastructure, not the model. The llama-server crashed mid-run and 5 tasks burned their retries against a dead port. On rerun with a stable server:
| Task | Initial | Retry |
|---|---|---|
| file-and-terminal-self-serve | fail (conn error) | PASS |
| web-search-required | fail (conn error) | PASS |
| use-a-skill | fail (conn error) | PASS |
| memory-recall | fail (conn error) | fail — raw score 1.0 but failed verification gate |
| delegate-parallel-subtasks | fail (conn error) | PASS |
| todo-plan | fail (conn error) | PASS |
| browser-required | fail (61 tool calls, wrong outcome) | — |
Corrected score: 11/13 (85%). For reference from our July sweep: Qwen3.5-9B UD Q4_K_XL scored 77% (with MTP spec-decoding, turbo KV, 262K ctx) and Qwen3.6-35B-A3B UD IQ3_XXS scored 85%. Caveats before declaring granite the winner: the Qwen runs used a different serving regime, and this suite is small enough that one task flips the percentage by 8 points. Treat as "granite G-Q5 is in the same tier as the best local models we've tested on this GPU," not a strict ranking.
<tool_call><function=name><parameter=...>); llama.cpp's --jinja parser handles it correctly — structured tool_calls come back properly."parallel_tool_calls": true in the request payload (llama.cpp default is off). Without it, only the first call in a turn survives.All measurements: AMD Ryzen 9 5950X, RTX 3060 12GB, Fedora 43, llama.cpp b062ba735, CUDA. Localmaxxing runs approved: G-Q5, run ids cmt8zlca200ioli019ukksygh (G-Q5), cmt8zl9b300ikli01o2v42b0q (Q4KS-XL), cmt8wx9zq00ftli014hldgmc6 (official Q4_K_M baseline).