FaceNet-Qwen3-8B-2Bit-Heretic

FaceNet-Qwen3-8B-2Bit-Heretic

Qwen3-8B abliterated with Heretic, quantized to 2-bit (IQ1_S) using a 30MB importance matrix for coherence preservation.

Heretic Abliteration

• Base model: Qwen/Qwen3-8B
• Method: Heretic (p-e-w/heretic), 20 Optuna trials, auto-selected best
• Refusal rate: 18/100 (down from ~99/100 baseline, 82% reduction)
• KL divergence: 0.112 (well within safe range)
• Abliteration applied to full-precision weights, then quantized

Quantization

• Format: IQ1_S (importance-weighted 2-bit)
• Bits per weight: 2.06 bpw
• Size: 2.0 GB (from 16.4 GB F16)
• Importance matrix: 30MB diverse corpus (Frankenstein + froggeric/imatrix + eaddario code/math/general English)
• Speed: ~95 tok/s on NVIDIA GB10 (Blackwell)

Variants

Usage

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="sidmishra/FaceNet-Qwen3-8B-2Bit-Heretic",
    filename="qwen3-8b-heretic-iq1_s.gguf",
    n_ctx=32768,
    n_gpu_layers=-1,
    verbose=False
)

response = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Your prompt here"}],
    max_tokens=200
)

Or with llama-server:

llama-server -m qwen3-8b-heretic-iq1_s.gguf --host 0.0.0.0 --port 8080 -ngl 99

Limitations

• Output uses reasoning_content field (Qwen3 thinking format)
• Occasional mild repetition artifacts at 2-bit (much rarer with the 30MB imatrix)
• For maximum quality, use the 3-bit or 5-bit variants

Acknowledgments

• Qwen/Qwen3-8B by Alibaba
• Heretic by p-e-w
• llama.cpp by GGML
• froggeric/imatrix - calibration backbone
• eaddario/imatrix-calibration - code + general
• Mary Shelley, for Frankenstein