Qwen3.5-9B-Uncensored-W4A16

Qwen3.5-9B-Uncensored-W4A16

We're exploring a new abliteration strategy and used the 9B model as a testbed. The result turned out well — this is currently the state-of-the-art abliterated 9B model across all three axes at once: capability retention, output brevity, and refusal removal. Three variants are published for different deployment needs — pick the one that matches your setup.

我们在尝试一个新的消融策略,以 9B 模型作为练兵场。效果不错 —— 这是目前 abliterated 9B 领域在能力保留、输出简洁、拒绝移除三方面同时达到最优的一版。 下面三个仓分别对应不同 的部署场景,按需选择即可。

This repo is the W4A16 quantization (INT4 weight + FP16 activation) of bowmanslayer/Qwen3.5-9B-Uncensored, 8.2 GB. Designed for vLLM serving with gptq_marlin kernel. Sister repos: full-precision bf16 in the main repo, llama.cpp GGUF (7 quant levels) in -GGUF.

本仓是 W4A16 量化版(INT4 权重 + FP16 激活),源自 bowmanslayer/Qwen3.5-9B-Uncensored, 8.2 GB,面向 vLLM 部署(gptq_marlin 快速核)。姐妹仓:主仓是全精度 bf16, llama.cpp GGUF(7 档)见 -GGUF。

Method: AutoRound → GPTQ format. Paired with fp8_e5m2 KV cache, capacity headroom is 60-80% larger than same-size competitors at the same GPU size.

方法:AutoRound → GPTQ 格式,vLLM gptq_marlin 快速核可用。 配合 fp8_e5m2 KV cache 拿到超同尺寸对手 60-80% 的上下文容量。

---

1. Quick start / 快速开始

Recommended vLLM launch (fp8 KV enabled for best context out of the box): 推荐 vLLM 启动(已开 fp8 KV,开箱最佳上下文):

vllm serve bowmanslayer/Qwen3.5-9B-Uncensored-W4A16 \
  --dtype float16 --tensor-parallel-size 2 \
  --kv-cache-dtype fp8_e5m2 \
  --max-model-len 28672 --max-num-seqs 16 \
  --reasoning-parser qwen3

Single-GPU variant / 单卡:

vllm serve bowmanslayer/Qwen3.5-9B-Uncensored-W4A16 \
  --dtype float16 --tensor-parallel-size 1 \
  --kv-cache-dtype fp8_e5m2 --gpu-memory-utilization 0.92

---

2. Full performance evaluation / 完整性能测试

Quantization did not break the ablation: 12 harmful quick-check on this W4A16 build recorded 0/10 refusals (text 0/6, image 0/4), consistent with bf16's 2/100 magnitude. All numbers below are the bf16 reference build (which this W4A16 was quantized from); W4A16-specific accuracy differences on a 3-task × 450-item spot check were within noise (abs delta +0.22).

量化不破消融:本 W4A16 版 12 条 harmful 快验记录到 0/10 拒绝(文字 0/6,图像 0/4),与 bf16 的 2/100 量级一致。以下所有数字取自本 W4A16 所派生的 bf16 参考版;W4A16 在 3 项 450 题抽测上与 bf16 精度差异在噪声范围(abs delta +0.22)。

2.1 Refusal rate / 拒绝率

100 harmful prompts from mlabonne/harmful_behaviors test split, 50 harmless controls from mlabonne/harmless_alpaca. Greedy decoding, 192 tokens, thinking disabled. Fixed judge model reads full response text (not keyword matching).

100 条 harmful 来自 mlabonne 数据集 test split,50 条 harmless 对照来自同源。 贪心解码,192 tokens,思考关闭。固定裁判读完整回复(非关键词匹配)。

2.2 11-benchmark capability comparison / 11 项能力对比

Same fixed seed, same harness, same items. All 4 models run under vLLM TP=2 + fp8_e5m2 KV + 24576 max output tokens, thinking enabled.

同一固定 seed,同 harness,同题,4 模型都在 vLLM TP=2 + fp8_e5m2 KV + 24576 max output tokens 下跑,thinking on。

Among the abliterated family (excluding base): 7 firsts out of 11, mean 76.14 leads Heretic-v2 (+3.5pp) and Huihui (+17.6pp).

在 abliterated 家族中(不算 base):7 项第一 / 11,均值 76.14 领先 Heretic-v2 3.5pp, 碾压 Huihui 17.6pp。

† TruthfulQA: serial-run contamination pulled it to 54, single-run isolation gives 62.68. Table uses the fair number. Behavioral drift not knowledge loss (see main repo §3).

† TruthfulQA:串行运行 vLLM 状态污染拉低到 54,单跑 62.68。表内取单跑。这是行为漂移 不是知识损伤(详见主仓 §3)。

2.3 Output brevity / 输出简洁度

This model is the only ablation more concise than base — median -17.7%, truncation -15%. Community ablations expand across the board (Heretic-v2 +66%, Huihui nearly 2×).

本模型是唯一比 base 更简洁的消融(中位 -17.7%,截断 -15%);社区版全线膨胀(Heretic-v2 +66%,Huihui 差点翻倍)。

---

3. Context capacity vs common ~8GB models / 和常见 ~8GB 版本的上下文对比

Qwen3.5-9B architecture = 32 layers (24 linear + 8 full-attention), hidden 3584. KV cache scales with context only on full-attention layers, so the numbers below are computed (excluding CUDA graph overhead etc., measured deviation <5%).

Qwen3.5-9B 架构 = 32 层(24 linear + 8 full attention),hidden 3584。KV cache 只对 full attention 层随 context 线性增长,因此本节数字是理论算(不含 CUDA graph overhead 等,实测偏差 <5%)。

Per-token KV cache size / 每 token KV cache 大小:

• fp16 KV: 2 (K+V) × 8 layers × 3584 × 2 bytes = 114 KB / token
• fp8 KV (this model default): 2 × 8 × 3584 × 1 = 57 KB / token (-50%)

After weight + ~1 GB overhead, remaining memory available for KV cache / 减去权重与 overhead 后,剩余可分配给 KV cache:

Takeaways / 要点:

• Our recommended launch defaults to fp8_e5m2 KV, no manual config needed 推荐启动默认就是 fp8_e5m2 KV,无需手动配置
• GGUF users need to manually add -ctk q8_0 -ctv q8_0 (int8 KV, similar effect) for equivalent context; most community guides omit this GGUF 用户要拿到同等上下文,需手动加 -ctk q8_0 -ctv q8_0(int8 KV,类似效果), 但大部分社区帖不给,默认体验只到我们一半
• The gap is largest on 12 GB cards (48k vs 30k, +60%) — the most tangible value differential on mid-tier hardware 12 GB 显卡的差异最大(+60% 上下文)—— 是这类中端卡上最有感知的价值差

---

4. Memory budget by scenario / 显存与量化档次对照

Using W4A16 weight 8.2 GB as baseline / 以 W4A16 权重 8.2 GB 为基准:

Deployment case (this is what we happen to run on, not a requirement): 2× RTX 3090 with TP=2 gives 1.22M-token KV cache headroom. Single 3090 works fine for typical serving. Any GPU with enough VRAM for the row you pick will work.

部署案例(这是我们自己的机型,非硬件要求): 2× RTX 3090 TP=2 可拿到 1.22M token KV cache 余量;单张 3090 常规服务也够用。任何显存够上表某一行的 GPU 都能跑。

---

5. Quantization recipe / 量化配方

AutoRound --bits 4 --group_size 128 --format auto_round:auto_gptq --nsamples 256 --seqlen 2048. linear_attn.in_proj_a/b (48 tensors, SSM computation, not quantizable) excluded. Vision tower and MTP head processed to int4 alongside language layers.

AutoRound --bits 4 --group_size 128 --format auto_round:auto_gptq --nsamples 256 --seqlen 2048。linear_attn.in_proj_a/b(48 张量,SSM 计算不可量化)排除。视觉塔 和 MTP 头随语言层一同处理为 int4。

MTP speculative decoding not recommended (vLLM 0.20.2: 40-46% slowdown, K=5 crash). MTP speculative decoding 不推荐启用(vLLM 0.20.2 实测减速 40-46%,K=5 崩溃)。

---

① Provenance and attribution / 来源与归属

Not an official Qwen release; not endorsed by or affiliated with the Qwen team or Alibaba Cloud.

非 Qwen 官方发布;与 Qwen 团队及阿里云无背书或从属关系。

---

② Safety alignment has been removed / 安全对齐已被移除

This is the point of the model, and you should read this before downloading. 这是模型的核心属性,下载前请务必阅读。

The upstream ablation removes the refusal behaviour trained into Qwen/Qwen3.5-9B. Automated refusal testing on the bf16 reference recorded 2/100 on 100 held-out adversarial prompts, versus 99/100 for the base. Quantization does not restore refusals (12-item spot check on this W4A16 build: 0/10).

上游消融移除了 Qwen/Qwen3.5-9B 训练进去的拒绝行为。bf16 参考版在 100 条 held-out 对抗 提示上自动测试记录到 2/100,而基座是 99/100。量化不恢复拒绝(本 W4A16 版 12 条 抽测:0/10)。

Consequences you are accepting / 你正在接受的后果:

• It will produce content the original model declines to produce, including content that is offensive, dangerous, or illegal in your jurisdiction. 会产生原模型拒绝产生的内容,包括在你所在司法辖区内可能属于冒犯性、危险或违法的内容
• It has no content filter. There is no safe-completion path, no refusal fallback, no guardrail to fail back to. 没有内容过滤;没有 safe-completion 路径,没有拒绝回落,没有护栏可依
• Refusal removal is not proven exhaustive — absence of observed refusals in testing is not proof that none remain, and equally not proof that no harmful behaviour was introduced. 拒绝移除未证明穷尽
• TruthfulQA drops 24 pp vs base — behavioral, not knowledge loss (see main repo). TruthfulQA 相对基座掉 24 pp —— 行为性,非知识损伤(见主仓)

Not intended for / 不适用于:

• Deployment to third parties without your own safety layer / 未搭建自己的安全层就部署给 第三方
• Anyone under the legal age in their jurisdiction / 未达所在司法辖区法定年龄的人
• Any use prohibited by Qwen's acceptable use policy / Qwen 可接受使用政策禁止的任何用途

Intended for / 适用于: local inference and research, by people who understand the above and take responsibility for it / 本地推理与研究,由理解以上内容并为其负责的用户使用。

---

③ No warranty; responsibility rests with the user / 无担保;责任在用户

This model is provided "AS IS", without warranty of any kind, express or implied, including but not limited to warranties of merchantability, fitness for a particular purpose, and non-infringement.

本模型按**"现状"提供,不提供任何明示或默示的担保**,包括但不限于对适销性、特定用途 适用性以及无侵权性的担保。

• I do not endorse, recommend, or condone any particular use of this model. 我不背书、不推荐、不认可对本模型的任何特定使用方式
• I make no representation that its outputs are accurate, lawful, or fit for any purpose. 我不对其输出的准确性、合法性或用途适用性作任何声明
• You are solely responsible for what you generate with it, for how you deploy it, and for compliance with all laws and regulations applicable to you — including but not limited to laws on illegal content, data protection, export control, and AI-specific regulation in your jurisdiction. 你独自承担全部责任:对生成内容、部署方式、以及遵守适用于你的一切法律法规负责
• To the maximum extent permitted by law, I accept no liability for any claim, damage, or other liability arising from the model or its use. 在法律允许的最大范围内,我不承担因本模型或其使用而产生的任何索赔、损害或其他责任

Downloading these files means you accept the above. If you do not, do not download them. 下载本仓文件即表示你接受以上内容。若不接受,请勿下载。

The author is not a lawyer; the above is not legal advice. 作者不是律师;以上不构成法律意见。

---

License

Apache 2.0, inherited through the chain above. The Apache-2.0 grant covers the weights; it does not grant permission for uses that are unlawful where you are.

Apache 2.0,依上述归属链继承。Apache-2.0 授权覆盖权重本身;不授权任何在你所在地不合法 的用途。