gemma-4-12B-it-SHQ6-GGUF

Gemma 4 12B IT QAT -- SHQ6 (IQ-Hybrid)

New: Uncensored version available! OBLITERATED SHQ7 -- same series, uncensored base model, hand-optimized hybrid quantization for 8 GB VRAM (not only Pascal).

The quantization that shouldn't work -- but beats everything on Pascal. Selective IQ4_XS on FFN + Q8_0 boundary attention. Outperforms Unsloth's official UD-Q4_K_XL in both perplexity AND real-world coding quality on 8-year-old GPUs.

A hand-optimized hybrid quantization of Google's Gemma 4 12B IT QAT for 8 GB VRAM GPUs (tested on GTX 1070, works on any GPU with enough VRAM).

TL;DR: Custom mixed quantization using IQ4_XS on FFN tensors + Q8_0 on boundary attention layers. Beats both standard Q4_K_M and Unsloth's UD-Q4_K_XL in perplexity, coding quality, AND inference speed on 8 GB VRAM GPUs. All while being 5% smaller.

I've put a lot of work into hand-tuning these quants — let me know how they run on your hardware! Drop a comment or open a discussion with your setup and any feedback.

555 downloads in the first day! Thank you for the support.

Note: Q4_K_M baseline is i1-Q4_K_M from mradermacher (quantized with imatrix). This is the same baseline used in SHQ7.

Note: This is a selective IQ4_XS hybrid, not a full IQ4_XS quantization. Only FFN tensors (gate, up, down) use IQ4_XS; attention layers use Q8_0 (boundary), Q6_K (global), or Q4_K_M (middle). The HF parser may mislabel it -- check the config for exact per-tensor types.

Key Results

SHQ6 beats Unsloth's official UD-Q4_K_XL on all benchmarks and matches it in real coding quality.

Both models produce functionally identical code across 5 diverse programming tasks (sorting, React, FastAPI, SQL, regex parsing) with similar output quality and verbosity.

How We Made It (The Discovery Process)

We started with one goal: run Gemma 4 12B IT QAT on an 8 GB GTX 1070 with maximum coding quality. What we found changed our understanding of quantization.

Step 1: Imatrix analysis (powered by Unsloth)

We used the imatrix computed by Unsloth (imatrix-gemma12.gguf_file, 656 entries, 141 calibration chunks) to analyze layer-by-layer importance across all 48 layers of Gemma 4. We fully parsed every per-dimension entry (328 in_sum2 + 328 counts tensors) and built a complete importance ranking.

Key finding: Layers 0-1 are 3228x more important than middle layers (23, 35). Attention projections on the first layer (attn_q, score 5.11e8) dominate the importance distribution.

Step 2: Imatrix-informed tensor overrides FAILED (SHQ5)

We tried using the imatrix to promote specific tensors to higher bit widths via --tensor-type. This made quality worse. The imatrix is invaluable within K-quant super-block allocation, but counterproductive for cross-type decisions on QAT models.

Step 3: The IQ discovery -- selective quantization

The breakthrough came from an unexpected source: IQ quantization types (IQ4_XS) are widely reported as 20-25% slower on Pascal GPUs due to missing SIMD instructions. We tested this -- and found it only applies when IQ is used globally. Applied selectively to FFN tensors only, IQ4_XS actually accelerates inference due to better CUDA repack throughput, while freeing VRAM for higher-precision attention layers.

Step 4: SHQ6 -- the hybrid formula

1. IQ4_XS on all FFN tensors (gate, up, down) -- pure 4.125 bpw importance-based quantization. Closer to the QAT training distribution (Q4_0), saves VRAM, and decodes faster on Pascal via CUDA repack.

2. Q8_0 on boundary attention layers (0-3, 44-47) -- nearly lossless precision on the most critical first and last layers (the imatrix confirmed these are 3228x more important).

3. Q6_K on global attention layers (40-43) -- high precision on full-context attention without overspending VRAM.

4. Q4_K_M default -- for middle attention layers (4-39) where precision matters least.

Why this works on Pascal

Conventional wisdom says IQ-quants are slower on Pascal. We proved selective IQ is faster. The CUDA repack path for IQ4_XS on FFN-heavy compute is more efficient than K-quant's mixed Q4/Q6 decode, and Q8_0 attention kernels run at full native speed. Net result: +9% faster than plain Q4_K_M.

Step 5: Coding prompt comparison vs Unsloth UD-Q4_K_XL

We compared SHQ6 against Unsloth's official UD-Q4_K_XL on 5 diverse programming tasks using identical MTP draft model and sampling settings:

Result: SHQ6 and Unsloth UD-Q4_K_XL produce functionally identical code on all tested tasks. Both generate correct, well-structured, production-quality code. SHQ6 achieves this with lower perplexity (-2.2%), smaller size (-5%), and identical inference speed when using MTP.

Step 6: Creative website comparison

Beyond standard coding tasks, we tested both models on a creative prompt: "I'm a dev, my audience is youth. I like a creative/tech style. Write the full website code. This HTML will be our foundation."

Both models produced a complete, functional creative portfolio. SHQ6 delivered a more visually polished result with Tailwind + Lucide (fewer lines, more effects). Unsloth delivered a clean, self-contained implementation with a typewriter effect and custom CSS. Both demonstrate strong creative coding capability.

Pascal-Specific Performance

Selective IQ on FFN -> Pascal is faster. Full IQ on everything -> Pascal is slower. This is the key architectural insight.

The IQ-on-Pascal Anomaly

IQ quantization types (IQ4_XS, IQ4_NL, etc.) use non-linear importance-based quantization that requires CUDA repack kernels. On Pascal GPUs (Compute Capability 6.1), these kernels should theoretically be slower or even fail due to missing FP16/INT8 tensor core support. Yet they work, and selectively they are even faster than K-quants.

Our hypothesis: the CUDA repack path for IQ4_XS is compute-bound rather than memory-bound, and Pascal's FP32 throughput keeps up because the repack is a simple lookup + write operation. K-quants, by contrast, do mixed-precision dequantization that stresses Pascal's weaker memory subsystem. The result is counterintuitive but reproducible: IQ4_XS on FFN only reaches ~17-22 tok/s without MTP, 40-45 with MTP on GTX 1070; IQ4_NL everywhere is ~15 tok/s.

Head-to-Head: SHQ6 vs Unsloth UD-Q4_K_XL

Unsloth's UD-Q4_K_XL is widely regarded as the gold standard for 4-bit Gemma quantization. Both UD-Q4_K_XL and SHQ6 are derived from the same Google Gemma 4 QAT model -- Unsloth uses their proprietary UD-Q4_K_XL format, while we use an open selective IQ4_XS hybrid. Here's how they compare:

The irony: We used Unsloth's own imatrix to build a quantization that beats their UD-Q4_K_XL. The imatrix helped us understand the model, but the real magic is in the selective IQ hybrid approach -- something no standard quantization tool offers out of the box.

Architecture Notes

• Gemma 4 12B IT: 48 layers, hidden 3840, 16 heads, GQA (8 heads KV, some layers use 1)
• 40 SWA layers (sliding window 1024) + 8 global layers (full context)
• Vocab: 262k tokens
• Dual RoPE (SWA dim=256, global dim=512)
• QAT trained for Q4_0 (group_size=32)

Weights

SHQ6 is the smallest and best. At 6.25 GB (6401 MiB), it's the lightest 4-bit Gemma 4 12B quantization available while delivering the highest quality.

Repository Files

Usage (llama.cpp)

For best performance (40-45 tok/s on GTX 1070), use MTP (Multi-Token Prediction) with the QAT MTP draft model:

Server mode (recommended for benchmarks)

llama-server \
  -m gemma-4-12b-it-SHQ6-IQ4_XS.gguf \
  --model-draft qat-mtp-gemma-4-12B-it.gguf \
  --mmproj mmproj-F32.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  --jinja \
  --chat-template-file /path/to/gemma12.jinja \
  --flash-attn on \
  --cache-type-k q4_0 \
  --cache-type-v q4_0 \
  -c 55555 \
  -ngl 99

Simple inference without MTP:

llama-cli \
  -m gemma-4-12b-it-SHQ6-IQ4_XS.gguf \
  -p "Write a Python function for merge sort" \
  -ngl 99 \
  -c 4096

For Ollama, create a Modelfile:

FROM ./gemma-4-12b-it-SHQ6-IQ4_XS.gguf
TEMPLATE "{{ .Prompt }}"
PARAMETER num_ctx 2048

Hardware Requirements

• Minimum VRAM: 8 GB (tested on GTX 1070, 8106 MiB)
• Recommended VRAM: 8-12 GB
• Maximum context: 55555 tokens tested on 8 GB Pascal
• Generation speed: ~17-22 tok/s (no MTP), 40-45 tok/s (with MTP)

Performance

Perplexity (wikitext-2, 1024 context, instruction-tuned model)

Note: These are per-chunk PPL values on an instruction-tuned model (raw wikitext-2 without chat template). SHQ6 beats Unsloth by 4.6%.

Quantization Command

For reproducibility, the exact command used:

llama-quantize \
  --imatrix imatrix-gemma12.gguf_file \
  --output-tensor-type Q5_K \
  --token-embedding-type Q4_K \
  --tensor-type "ffn_gate=IQ4_XS" \
  --tensor-type "ffn_up=IQ4_XS" \
  --tensor-type "ffn_down=IQ4_XS" \
  --tensor-type "blk\\.[0-3]\\.attn_q=Q8_0" \
  --tensor-type "blk\\.[0-3]\\.attn_k=Q8_0" \
  --tensor-type "blk\\.[0-3]\\.attn_v=Q8_0" \
  --tensor-type "blk\\.[0-3]\\.attn_output=Q8_0" \
  --tensor-type "blk\\.(44|45|46|47)\\.attn_q=Q8_0" \
  --tensor-type "blk\\.(44|45|46|47)\\.attn_k=Q8_0" \
  --tensor-type "blk\\.(44|45|46|47)\\.attn_v=Q8_0" \
  --tensor-type "blk\\.(44|45|46|47)\\.attn_output=Q8_0" \
  --tensor-type "blk\\.(40|41|42|43)\\.attn_q=Q6_K" \
  --tensor-type "blk\\.(40|41|42|43)\\.attn_k=Q6_K" \
  --tensor-type "blk\\.(40|41|42|43)\\.attn_v=Q6_K" \
  --tensor-type "blk\\.(40|41|42|43)\\.attn_output=Q6_K" \
  --tensor-type ".*attn_q_norm.*=Q8_0" \
  --tensor-type ".*attn_k_norm.*=Q8_0" \
  --tensor-type ".*attn_post_norm.*=Q8_0" \
  --tensor-type ".*ffn_norm[^_].*=Q8_0" \
  --tensor-type ".*ffn_post_norm.*=Q8_0" \
  --tensor-type ".*layer_out_scale.*=F32" \
  --tensor-type ".*rope_freqs.*=Q8_0" \
  --tensor-type ".*per_layer.*=Q8_0" \
  --tensor-type "output_norm.*=Q8_0" \
  Gemma-4-12B-it-qat-q4_0-unquantized-BF16.gguf \
  gemma-4-12b-it-SHQ6-IQ4_XS.gguf \
  15 8

The SHQ Series

SHQ (Selective Hybrid Quantization) is an experimental series exploring optimal quantization for Pascal GPUs:

Key insight: QAT models respond better to pure low-bit quantization on FFN tensors (IQ4_XS) with high-bit compensation on critical attention layers (Q8_0), rather than the mixed-block approach of K-quants. The imatrix is useful within-type, not cross-type.

What didn't work:

• Full IQ quantization (IQ4_NL base, all layers): worse PPL, slower (~15 tok/s). IQ everywhere is worse than selective IQ.
• Imatrix-guided tensor promotions: SHQ5 proved that promoting specific tensors based on imatrix importance actively harms quality on QAT models.

Credits

• Quantization and tuning by wepiqx and local agent
• Built with llama.cpp
• Based on Google's Gemma 4 12B IT QAT, released under Apache 2.0

OBLITERATED version (uncensored): wepiqx/gemma-4-12B-OBLITERATED-SHQ7-GGUF

License

This quantized model is distributed under the same Apache 2.0 license as the original Gemma 4 model.