Granite-4.0-H-350M-Unsloth-MagicQuant-Hybrid-GGUF

MagicQuant GGUF Hybrids - granite 4.0 h 350m unsloth

(DEPRECIATED - Part of MagicQuant v1.0 which had significant flaws. Please utilize v2.0 which is production ready)

MagicQuant is an automated quantization, benchmarking, and evolutionary hybrid-GGUF search system for LLMs.

Each release includes models optimized to outperform standard baseline quants (Q8, Q6, Q5, Q4). If a baseline GGUF exists in this repo, the evolutionary engine couldn’t beat it. If a baseline is missing, it’s because a hybrid configuration outperformed it so completely that including the baseline would've been pointless.

These hybrid GGUFs are built to be as small, fast, and low-drift as possible while preserving model capability.

To dive deeper into how MagicQuant works, see the main repo: MagicQuant on GitHub (by MagicCodingMan)

Notes:

• The HuggingFace hardware compatibility where it shows the bits is usually wrong. It doesn't understand hybrid mixes, so don't trust it.
• Naming scheme can be found on the MagicQuant Wiki.
• (tips) Less precision loss means less brain damage. More TPS means faster! Smaller is always better right?

Precision Loss Guide

• 0–0.1% → God-tier, scientifically exact
• 0.1–1% → True near-lossless, agent-ready
• 1–3% → Minimal loss, great for personal use
• 3–5% → Borderline, but still functional
• 5%+ → Toys, not tools, outside MagicQuant’s scope

Learn more about precision loss here.

IMPORTANT NOTE: Due to this model being so small. The test was significantly stricter in what precision loss was allowed.

Table - File Size + TPS + Avg Precision Loss

Table - PPL Columns

• gen = ppl_general, code = ppl_code, math = ppl_math

Table - Precision Loss Columns

• loss_* values are absolute precision-loss % vs BF16 per domain.

---

Baseline Models (Reference)

Table - File Size + TPS + Avg Precision Loss

Table - PPL Columns

• gen = ppl_general, code = ppl_code, math = ppl_math

Table - Precision Loss Columns

• loss_* values are absolute precision-loss % vs BF16 per domain.

Support

I’m a solo developer working full time for myself to achieve my dream, pouring nights and weekends into open protocols and tools that I hope make the world a little better. If you chip in, you're helping me keep the lights on while I keep shipping.

Click here to see ways to support - BTC, Paypal, GitHub sponsors.

Or, just drop a like on the repo :)