Downloads · 30 days
86
15% of all-time downloads
nguyenthilaitrieulong/gemma-4-31B-it-heretic-Q8_K_XL-GGUF
gemma-4-31B-it-heretic-Q8_K_XL-GGUF is a image-text-to-text model from nguyenthilaitrieulong. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for gguf. The card lists the license as gemma.
Imatrix-calibrated Q8KXL GGUF of coder3101/gemma-4-31B-it-heretic, built for llama.cpp. Includes the full BF16 vision projector so the model retains image understanding.
Downloads · 30 days
86
15% of all-time downloads
All-time downloads
562
Public
Repo size
36.2 GB
Likes
1
Public
Click a slice to open those files.
.gguf36.2 GB · 100%
From the Hugging Face model README
Imatrix-calibrated Q8_K_XL GGUF of coder3101/gemma-4-31B-it-heretic, built for llama.cpp. Includes the full BF16 vision projector so the model retains image understanding.
Heretic is an abliterated variant of google/gemma-4-31B-it that suppresses refusal behavior while preserving instruction-following. The abliteration survives quantization cleanly — post-quant tests show the model still answers directly without hedging.
| File | Size | Notes |
|---|---|---|
gemma-4-31B-it-heretic-Q8_K_XL.gguf | ~33 GB | Language model weights (Q8_0 base + selective F16) |
mmproj-BF16.gguf | ~1.2 GB | Vision projector, full BF16 |
Base format is Q8_0 with the token embedding and output tensors also at Q8_0. Six transformer blocks identified as activation-outlier hotspots are escalated to F16 on their attention Q/K and full MLP (gate / up / down):
blk.{1,52,53,57,58,59}.attn_q.weight = f16
blk.{1,52,53,57,58,59}.attn_k.weight = f16
blk.{1,52,53,57,58,59}.ffn_gate.weight = f16
blk.{1,52,53,57,58,59}.ffn_up.weight = f16
blk.{1,52,53,57,58,59}.ffn_down.weight = f16
This mirrors the bit-allocation pattern used by Unsloth's UD-Q8_K_XL variants: protect the first transformer block and the last few blocks where activation magnitudes blow up, leave the rest at Q8_0 where the K-quant superblock structure handles things just fine.
Imatrix was computed from a 613K-token mixed corpus:
bartowski/c4_datasets_calibration_data calibration_datav3.txt — general text, ~70K tokenseaddario/imatrix-calibration tools_micro.parquet — tool-calling patterns, ~400K tokenseaddario/imatrix-calibration code_micro.parquet — code instructions, ~150K tokensImatrix run: 282 chunks at --ctx 2048 --batch 512, full GPU offload, on 4× A100 80GB.
Gemma 4's MLP blocks have heavy-tailed activation distributions. GPTQ's per-column Hessian update fails Cholesky inversion on every down_proj even with aggressive dampening, falls back to RTN, and produces visible token-level corruption in code output (background-color: #3-, dropped articles, etc.) at every bit width tested (W4A16 sym/asym, W8A16, mixed). llama.cpp's K-quant superblock-scale format plus per-channel imatrix weighting handles the outliers correctly. Validated with code generation, reasoning, and tool-calling tests post-quant.
llama-server \
-m gemma-4-31B-it-heretic-Q8_K_XL.gguf \
--mmproj mmproj-BF16.gguf \
-ngl 999 \
-c 262144 \
--flash-attn on \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--jinja \
--chat-template-kwargs '{"enable_thinking":true}' \
--temp 1.0 --top-p 0.95 --top-k 64 \
--host 0.0.0.0 --port 8081
Tested on a 4× 24 GB rig (1× RTX 4090 + 3× RTX 3090) with 256K context, q8_0 KV cache, flash attention on. The Q8_0 KV cache is the only thing that makes the long context fit.
Q8_K_XL label follows Unsloth's naming convention for "Q8 base with selected layers escalated to higher precision" — the actual ggml type of the base tensors is q8_0.google/gemma-4-31B-it — base modelcoder3101/gemma-4-31B-it-heretic — abliteration