FaceNet-Qwen3-8B-5Bit-Heretic

FaceNet-Qwen3-8B-5Bit-Heretic

Qwen3-8B abliterated with Heretic, quantized to 5-bit (Q4_K_M) — the practical sweet spot between size and quality.

Heretic Abliteration

• Base model: Qwen/Qwen3-8B
• Method: Heretic (p-e-w/heretic), 20 Optuna trials, auto-selected best
• Refusal rate: 18/100 (down from ~99/100 baseline, 82% reduction)
• KL divergence: 0.112
• Abliteration applied to full-precision weights, then quantized F16 → Q4_K_M

Quantization

• Format: Q4_K_M (K-quant mixture: attention layers ~2-bit, FFN ~5-bit)
• Bits per weight: 4.90 bpw
• Size: 4.8 GB (from 16.4 GB F16)
• Speed: ~42 tok/s on NVIDIA GB10 (Blackwell)
• Coherence: Perfect — no artifacts

Variants

This 5-bit variant is the recommended default — excellent balance of speed, size, and quality.

Usage

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="Ontologer/FaceNet-Qwen3-8B-5Bit-Heretic",
    filename="qwen3-8b-heretic-q4_k_m.gguf",
    n_ctx=32768,
    n_gpu_layers=-1,
    verbose=False
)

Acknowledgments

• Qwen/Qwen3-8B by Alibaba
• Heretic by p-e-w
• llama.cpp by GGML