Qwen3.8-27B-heretic-SmoothQuant-W8A8-INT8

Qwen3.8-27B — Heretic Abliterated · Rotation + SmoothQuant + GPTQ · W8A8 INT8

[!CAUTION] Research artifact. Safety alignment has been deliberately removed. This model will attempt to comply with harmful, dangerous, illegal, and unethical requests that the base model refuses, with no content moderation. See Safety. Provided for research purposes only, with no warranty and no liability accepted by the author — see the disclaimer at the bottom.

An INT8 W8A8 quantization of Qwen3.8-27B-heretic (the abliterated BF16 build of Qwen/Qwen3.8-27B), for fast serving on Ampere-class (SM 8.0) GPUs where native FP8 is unavailable.

• Weights: INT8, per-channel symmetric (GPTQ).
• Activations: INT8, per-token dynamic → compressed-tensors int-quantized, native CUTLASS INT8 tensor-core path.
• Left unquantized (FP16): the MTP speculative-decoding head, lm_head, embed_tokens, the vision tower, GatedDeltaNet recurrent gates, all norms. Note these are FP16, not the source's BF16 — the 16-bit residual precision of this build is float16 throughout (config.json: "dtype": "float16").
• Recipe: offline QuaRot rotation → SmoothQuant → GPTQ (llm-compressor).

The MTP head is unquantized but is not byte-identical to the BF16 build's: it carries the same residual-stream rotation and RMSNorm fold as the rest of the model, so the two heads are not interchangeable. Its RMSNorm weights are stored as zeros, which is the identity scale under Qwen3.5's (1 + weight) RMSNorm.

This model vs. its source

Source: Qwen3.8-27B-heretic (the abliterated BF16). This build changes numerics only; refusal behaviour is identical to the source (0/100 hard refusals; see the source card for the ~20–24 % chain-of-thought deflection note).

Quantization fidelity — KL divergence from the source, KL(source ‖ this), measured served under vLLM (real INT8 dynamic activations) on WikiText-2, top-512, 12 264 token positions:

On par with the reference W8A8 of this model — the abliteration does not make the weights harder to quantize.

Speculative decoding — acceptance, HumanEval, concurrency 32, vLLM (fp8_e4m3 KV, async scheduling, MTP num_speculative_tokens=3):

Acceptance length (mean tokens accepted per verify step) sets the speculative speedup; it is unchanged from the non-abliterated model. Heretic leaves the MTP head untouched — it is bitwise identical to the source's — and the abliteration is a low-KL edit (0.088 nats) confined to 63 output projections in layers 11–63, so drafter/target agreement barely moves.

Serving (vLLM)

Target: 2× Ampere (SM 8.0).

vllm serve akumaburn/Qwen3.8-27B-heretic-SmoothQuant-W8A8-INT8 \
  --tensor-parallel-size 2 \
  --max-model-len 262144 \
  --kv-cache-dtype fp8_e4m3 \
  --async-scheduling \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

All quantized matmul dimensions are multiples of 128 (no INT8 padding); TP=2 shards cleanly (GQA kv_heads=4 divides evenly).

Safety

Refusal behaviour has been deliberately removed; this model produces content the source declines, including dangerous, illegal, or unethical material, with no moderation. Intended for interpretability/safety research, red-teaming, and evaluation by people who understand and accept those risks. Do not deploy it where it can reach people who have not consented to unfiltered output. You are responsible for your use and for compliance with all applicable laws.

Disclaimer

This model is provided for research purposes only, "AS IS", without warranty of any kind, express or implied. The author accepts no liability for any use of this model or any consequences arising from it. By downloading or using it, you accept sole responsibility for your use and for compliance with all applicable laws and regulations. Base model © Qwen (Apache-2.0), inherited by this derivative; abliteration © the Heretic project; quantization via llm-compressor / compressed-tensors.