Llama-3-8B-Instruct-DeepRefusal-Broken

Llama-3-8B-Instruct-DeepRefusal-Broken

DeepRefusal's refusal direction defense, broken by abliterix — where every other public attack failed.

This model is produced from skysys00/Meta-Llama-3-8B-Instruct-DeepRefusal, the defended release accompanying "Beyond Surface Alignment: Rebuilding LLMs Safety Mechanism via Probabilistically Ablating Refusal Direction" (arXiv:2509.15202, EMNLP 2025 Findings, Xie et al.).

The DeepRefusal paper is explicit about its claims:

[2026/04/09] We evaluated heretic, presently the most prominent LLM censorship removal tool, and discovered—somewhat unexpectedly—that our approach exhibits strong resilience against such attacks. Adversaries appear unable to circumvent the model's built-in safety guardrails without triggering severe performance collapse.

abliterix falsifies this. 89% ASR. 14/15 hardcore prompts compliant. Zero fine-tuning. Two script invocations.

---

Head-to-head: abliterix vs every prior attack

Every attack the DeepRefusal paper tested, plus the tool they specifically called out by name (heretic), lands in the 0–2 % range. abliterix lands at 89 % on the same model, evaluated under a stricter judge (Gemini 3.1 Flash Lite LLM judge vs. the paper's in-house classifier).

Benchmark numbers

The 15 hardcore prompts cover lock-picking, phishing templates, methamphetamine synthesis, WiFi hacking, fake news, hotwiring, pipe-bomb construction, signature forgery, credential-stealing malware, shoplifting, and their Chinese-language equivalents. A single prompt (Chinese network-fraud how-to) produced a partial compliance that drifted into prevention advice — the remaining 14 are unambiguous.

How it was broken

DeepRefusal is a merged rank-16 LoRA adapter on top of Meta-Llama-3-8B-Instruct. SVD of W_defended − W_base confirms this: v_proj, o_proj, and gate_proj all show a clean singular-value cliff at rank 16, exactly matching the paper's published lora_rank = 16 hyperparameter.

Given that, the attack is:

1. Attenuate the LoRA delta. Compute W' = W_base + λ · (W_defended − W_base) with λ = 0.3. This halves the strength of DeepRefusal's "rebuild refusal from backup pathways" circuitry without restoring the base model.
2. Standard single-direction abliteration on the attenuated weights. mean vector method, direct steering mode, projected_abliteration = true, 60 Optuna trials over 4 attention/MLP components.

That's it. No iterative subspace tricks, no multi-direction SVD, no gradient optimization, no fine-tuning. Just weight arithmetic plus the standard abliteration loop that heretic and abliterix have used from day one.

Why the defense fails. DeepRefusal's training loss (paper §4.2–4.3, eq. 7) explicitly defends against h − r̂r̂ᵀh interventions applied at inference time. It teaches the model to keep the refusal direction in the residual stream even when that specific projection is applied. It does not defend against the training-time weights themselves being mechanically attenuated — that is out of distribution in both weight space and intervention space. Once the LoRA delta's effective magnitude drops, the rebuilt-refusal machinery falls below the threshold at which a standard linear abliteration can find and subtract it.

Reproduction (~2 hours on a single RTX 6000 Ada)

git clone https://github.com/wuwangzhang1216/abliterix
cd abliterix && pip install -e .

# Step 1: attenuate the LoRA delta
python scripts/deeprefusal_attenuate.py \
    --base NousResearch/Meta-Llama-3-8B-Instruct \
    --defended skysys00/Meta-Llama-3-8B-Instruct-DeepRefusal \
    --output ./llama3_dr_attenuated \
    --lambda 0.3

# Step 2: standard abliteration on the attenuated weights
AX_CONFIG=configs/llama3_8b_deeprefusal_attenuated.toml abliterix

# Step 3: export the best trial
python scripts/export_model.py \
    --model ./llama3_dr_attenuated \
    --checkpoint checkpoints_llama3_dr_attenuated \
    --trial 52 \
    --config configs/llama3_8b_deeprefusal_attenuated.toml \
    --push-to YOUR_USER/Llama-3-8B-Instruct-DeepRefusal-Broken

Full write-up and discussion: abliterix issue #11.

Why abliterix beats heretic here (and elsewhere)

abliterix is a direct derivative of heretic that has kept adding ammunition while the problem got harder. The DeepRefusal attack is built out of features heretic does not ship:

• Weight-delta attenuation (scripts/deeprefusal_attenuate.py) — needed the moment a defender merges a LoRA adapter into the base model to hide it.
• Direct weight projection mode with optional projected abliteration, discriminative layer selection, and norm-preserving updates — the combination that makes the final abliteration step work at low KL on the attenuated model.
• LLM-judge + LoRA + Gemini pipeline in the Optuna loop, so every trial is graded by a capable classifier rather than keyword matching, avoiding the false-positive inflation that plagues most abliteration leaderboards.
• 150+ pre-built model configs across dense, MoE, SSM/hybrid, and VL architectures — so when a novel defense drops, the turnaround from "new HF release" to "running benchmark" is one command.
• HonestAbliterationBench — a frozen evaluation contract (min_new_tokens=100, max_new_tokens=150, greedy, LLM judge, KL vs declared base) that resists the two failure modes (short generations + keyword judges) that make most abliteration numbers meaningless. DeepRefusal's own ASR claims hold up under keyword matching and collapse under LLM-judge scoring — we re-ran their baseline under both.

Same author family, same lineage, stronger toolbox.

Intended use and safety

This is a red-team artifact. It exists to demonstrate that the defense published in arXiv:2509.15202 does not generalize against the weight-space attacks that representation-engineering tools have been using for over a year.

Do not deploy this model in user-facing products. Do not use it to generate content that is illegal in your jurisdiction. If you are a safety researcher and you want to cite the result, please also cite the DeepRefusal paper and note the specific commit of abliterix used.

Credits

• Base model: Meta AI — meta-llama/Meta-Llama-3-8B-Instruct (via the NousResearch mirror for the delta computation).
• Defended base: Xie et al. — skysys00/Meta-Llama-3-8B-Instruct-DeepRefusal, arXiv:2509.15202.
• Tooling: abliterix, a derivative of heretic by Philipp Emanuel Weidmann. DeepRefusal attack pipeline landed in commit ac2197c.
• Author: Wangzhang Wu (@wuwangzhang1216).

<!-- wangzhang-provenance-license-v1 -->

Provenance and Modification Notice

• Immediate source checkpoint: skysys00/Meta-Llama-3-8B-Instruct-DeepRefusal
• Ultimate upstream model: meta-llama/Meta-Llama-3-8B-Instruct
• Exact base revision used: Not recorded in the existing release artifacts; the current upstream HEAD is not substituted.
• Modification method: Abliterix weight-space / representation intervention intended to reduce refusal behavior.
• Modified and published by: Wangzhang Wu
• Repository first published: 2026-04-13 (Hugging Face repository metadata)

The original model weights and/or derived checkpoint were modified. This repository is an independent derivative and is not an official release of the upstream model developer.

License and Attribution

The governing upstream license is Meta Llama 3 Community License. A copy is included in LICENSE. License source audited on 2026-08-29: https://github.com/meta-llama/llama-models/blob/main/models/llama3/LICENSE

All applicable upstream copyright, attribution, acceptable-use, and other license terms remain in effect. This repository grants no rights beyond those provided by the upstream license. Downstream users must preserve applicable license and attribution notices.

Built with Meta Llama 3. Use and redistribution remain subject to the Meta Llama 3 Community License and its incorporated Acceptable Use Policy.

---

<!-- wangzhang-safety-disclaimer-v1 -->

Disclaimer and Responsible Use / 免责声明与安全使用声明

English

This is an experimental, modified model provided for research, evaluation, and other lawful purposes. Its safety alignment, refusal behavior, or other safeguards may have been weakened or removed. It may produce inaccurate, biased, offensive, explicit, dangerous, or illegal content. Outputs are not professional advice and must not be relied on for medical, legal, financial, safety-critical, or other high-stakes decisions without qualified human review.

You are solely responsible for how you access, use, deploy, fine-tune, or redistribute this model and its outputs, including compliance with applicable laws, regulations, licenses, third-party rights, platform policies, and the original model's terms. Do not use it to facilitate harm, illegal activity, malware, fraud, privacy violations, targeted harassment, weapons development, or decisions that materially affect a person's rights or access to essential services without appropriate authorization, safeguards, and qualified oversight.

Before deployment, perform a context-specific risk assessment and testing; use human oversight, access controls, content filtering, rate limits, monitoring, logging, and incident-response procedures as appropriate. Preserve this notice in downstream redistributions.

The model is provided "AS IS", without warranties of any kind. To the fullest extent permitted by applicable law, the maintainer disclaims liability for claims, damages, or losses arising from use, misuse, inability to use, or redistribution of the model or its outputs. Nothing in this notice overrides applicable law or the governing license, and this notice is not legal advice.

中文

本模型属于实验性改造模型，仅供研究、评测及其他合法用途。其安全对齐、拒答机制或其他防护可能已被削弱或移除，因此可能生成不准确、偏见、冒犯、露骨、危险或违法内容。输出不构成医疗、法律、金融等专业意见；涉及高风险或重大权益的决定，必须由具备资质的人员复核。

使用者须对模型及其输出的访问、使用、部署、微调和再分发承担全部责任，并遵守适用法律法规、许可证、第三方权利、平台政策及原模型条款。不得将本模型用于促成伤害、违法活动、恶意软件、欺诈、侵犯隐私、定向骚扰、武器开发，或在缺乏适当授权、防护和专业监督时，用于实质影响个人权利或基本服务获取的决策。

部署前应进行与具体场景相匹配的风险评估和测试，并酌情采用人工监督、访问控制、内容过滤、限流、监控、日志和事件响应措施；下游再分发时应保留本声明。

本模型按“现状”提供，不附带任何形式的保证。在适用法律允许的最大范围内，维护者不对因使用、误用、无法使用或再分发本模型及其输出而产生的索赔、损害或损失承担责任。本声明不取代适用法律或管辖本模型的许可证，也不构成法律意见。