Qwen3-4B-Instruct-2507-abliterated-FP8

Qwen3-4B-Instruct-2507 — Abliterated, FP8 (W8A8)

A research artifact: the "refusal direction" (Arditi et al. 2024, arXiv:2406.11717) removed from Qwen/Qwen3-4B-Instruct-2507, then quantized to FP8. Its purpose is to measure the quality cost of abliteration with paired statistics on held-out data — a number the ecosystem of "uncensored" uploads reports for essentially no one.

If you want a safety-aligned assistant, use the base model. This is an instrument for studying what abliteration does to a model, published with the measurements that justify that framing.

What was done

Closed-form, no training:

1. Last-token residual activations collected for 128 refused prompts (AdvBench) vs. 128 harmless prompts (Alpaca), per layer.
2. refusal_dir = normalize(mean(harmful) − mean(harmless)).
3. Candidate layers swept by measured held-out refusal rate; layer 24 (depth 0.67 of 36) won.
4. Every residual-writing matrix orthogonalized against that direction (embed_tokens, per-layer o_proj and mlp.down_proj): W ← W − r̂(r̂ᵀW).
5. Re-quantized to FP8 (FP8_DYNAMIC, per-channel weights + per-token dynamic activations, lm_head excluded) — identical recipe to the non-abliterated FP8 sibling.

Pipeline and eval code: https://github.com/Subalzt/qwen3-4b-quantized-fp8-gptq

Measured cost (paired, vs. the non-abliterated FP8)

Harness validated — the non-abliterated FP8 reproduced PPL 10.0415 to the digit.

The two capability metrics disagree, and that is the finding. A functional code eval sees no significant loss; a dense per-token perplexity measure sees a large, unambiguous degradation on every chunk. The +19.5% perplexity cost is roughly 6× the +3.04% cost of 4-bit GPTQ quantization on the same model — abliteration is a larger quality intervention than quantization itself.

Intended use

Research on safety-tuning mechanisms, refusal-direction interpretability, quantization × behavioral-edit interaction, and robustness evaluation.

Limitations & responsible use

• Refusal behavior is substantially reduced by design. It is not safety- aligned and will comply with many requests the base model declines. Deploy only with your own safeguards and within applicable law and the Apache-2.0 terms inherited from the base model.
• General language-modeling quality is measurably degraded (+19.5% PPL); do not treat it as quality-equivalent to the base.
• FP8 (compressed-tensors) runs under vLLM / transformers on Linux. For native Windows use a GGUF build instead.

Run (vLLM)

vllm serve Subalzt/Qwen3-4B-Instruct-2507-abliterated-FP8 --max-model-len 8192

Citation

Arditi et al., Refusal in Language Models Is Mediated by a Single Direction, 2024. Base model © Qwen, Apache-2.0.