Qwen3.5-0.8B-abliterated

Qwen3.5-0.8B Abliterated (Uncensored)

This is an abliterated (uncensored) version of Qwen/Qwen3.5-0.8B.

Abliteration removes the model's refusal behavior by identifying and subtracting the "refusal direction" in the model's residual stream. This implementation is inspired by Arditi et al. 2024 ("Refusal in Language Models Is Mediated by a Single Direction") but extends the original method with several improvements from subsequent work.

Method

This model uses an enhanced multi-layer abliteration pipeline that goes beyond the original single-layer, PCA-based approach of Arditi et al. 2024. The key differences from the original paper and the additional techniques used are documented below.

Differences from Arditi et al. 2024

Additional Techniques Used

• Multi-layer intervention: Modifies 12 layers instead of 1 to counter the Hydra Effect (McGrath et al. 2023), where removing the refusal direction at a single layer can cause other layers to compensate.
• Projected refusal direction: Decomposes the refusal direction into a component parallel to the harmless mean (preserved) and a component orthogonal to it (removed). This targets only the refusal-specific signal while preserving harmless capabilities. Based on Lai (Oct 2025, "Projected Abliteration") and Zhao et al. 2025.
• Norm-preserving weight modification: After ablating the refusal direction from weight matrices, each row is rescaled to its original Frobenius norm to prevent capability degradation. Based on Lai (Nov 2025, "MPOA") and DoRA (Liu et al. 2024).
• Winsorization: Activation magnitudes are capped at the 0.995 quantile before computing means, preventing GeGLU outliers from destabilizing the refusal direction estimate. Based on Lai (Mar 2026, "ORBA").
• Double Gram-Schmidt orthogonalization: Applied twice for numerical stability (Horning et al. 2020).
• Composite layer selection: Layers are ranked by SNR * (1 - cosine_similarity) between harmful and harmless activation means, selecting layers where the refusal signal is strongest and most distinct from general capabilities.

Pipeline Steps

1. Scan: Collect last-token residual stream activations on 128 harmful vs 128 harmless prompts across layers 2-21 (skipping first 2 and last 2 of 24 total layers).
2. Compute: For each scanned layer, compute the mean difference between harmful and harmless activations, winsorize, project orthogonal to harmless mean, and compute quality metrics (SNR, cosine dissimilarity).
3. Select: Rank layers by composite score and select the top 12 (50% of 24 layers).
4. Abliterate: For each selected layer, project out the per-layer refusal direction from all 7 weight matrices (Q, K, V, O projections + gate, up, down MLPs), then rescale rows to preserve original Frobenius norms.
5. Save: Save the modified weights as a standard HuggingFace model.

Total matrices modified: 48 (4 attention + 3 MLP per layer, across 12 layers).

Abliteration Configuration

Evaluation Results

Comprehensive evaluation on 100 harmful prompts (from mlabonne/harmful_behaviors), 20 harmless prompts, and standard benchmarks. Both the base and abliterated models were tested on the same prompts for a fair comparison.

Refusal Behavior (Base vs Abliterated)

Refusal by Category (Base vs Abliterated)

Language Modeling Quality

Benchmark Comparison (Zero-Shot)

The enhanced abliteration causes only -1.00 percentage point degradation on both benchmarks, demonstrating minimal capability loss while reducing refusal rate from 86% to 3%.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "lew96123/Qwen3.5-0.8B-abliterated"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype="auto", device_map="auto"
)

messages = [{"role": "user", "content": "Tell me about the history of cryptography"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Disclaimer

This model has been modified to reduce refusal behavior. It may generate content that the original model would refuse. Use responsibly and in accordance with applicable laws and ethical guidelines. The creator is not responsible for any misuse.

References

@article{arditi2024refusal,
  title={Refusal in Language Models Is Mediated by a Single Direction},
  author={Arditi, Andy and Obeso, Oscar and Syed, Aaquib and Paleka, Daniel and Panickssery, Nina and Guo, Wes and Nandi, Hrishav},
  journal={arXiv preprint arXiv:2406.11717},
  year={2024}
}

@article{mcgrath2023hydra,
  title={The Hydra Effect: Emergent Self-repair in Language Model Computations},
  author={McGrath, Thomas and Rahtz, Matthew and Kramar, Janos and Mikulik, Vladimir and Legg, Shane},
  journal={arXiv preprint arXiv:2307.15771},
  year={2023}
}

@article{liu2024dora,
  title={DoRA: Weight-Decomposed Low-Rank Adaptation},
  author={Liu, Shih-Yang and Wang, Chien-Yi and Yin, Hongxu and Molchanov, Pavlo and Wang, Yu-Chiang Frank and Cheng, Kwang-Ting and Chen, Min-Hung},
  journal={arXiv preprint arXiv:2402.09353},
  year={2024}
}