FaceNet-Qwen3-8B-3Bit-Heretic

FaceNet-Qwen3-8B-3Bit-Heretic

Qwen3-8B abliterated with Heretic, quantized to 3-bit (Q2_K) — the practical sweet spot between size and quality.

Heretic Abliteration

• Base model: Qwen/Qwen3-8B
• Method: Heretic (p-e-w/heretic), 20 Optuna trials, auto-selected best
• Refusal rate: 18/100 (down from ~99/100 baseline, 82% reduction)
• KL divergence: 0.112
• Abliteration applied to full-precision weights, then quantized F16 → Q2_K

Quantization

• Format: Q2_K (K-quant mixture: attention layers ~2-bit, FFN ~3-bit)
• Bits per weight: 3.20 bpw
• Size: 3.1 GB (from 16.4 GB F16)
• Speed: ~58 tok/s on NVIDIA GB10 (Blackwell)
• Coherence: Perfect — no artifacts

Variants

This 3-bit variant is the recommended default — excellent balance of speed, size, and quality.

Usage

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="Ontologer/FaceNet-Qwen3-8B-3Bit-Heretic",
    filename="qwen3-8b-heretic-q2_k.gguf",
    n_ctx=32768,
    n_gpu_layers=-1,
    verbose=False
)

Acknowledgments

• Qwen/Qwen3-8B by Alibaba
• Heretic by p-e-w
• llama.cpp by GGML