Qwen3.5-9B-Uncensored-GGUF

Qwen3.5-9B-Uncensored-GGUF

We're exploring a new abliteration strategy and used the 9B model as a testbed. The result turned out well — this is currently the state-of-the-art abliterated 9B model across all three axes at once: capability retention, output brevity, and refusal removal. Three variants are published for different deployment needs — pick the one that matches your setup.

我们在尝试一个新的消融策略,以 9B 模型作为练兵场。效果不错 —— 这是目前 abliterated 9B 领域在能力保留、输出简洁、拒绝移除三方面同时达到最优的一版。 下面三个仓分别对应不同 的部署场景,按需选择即可。

This repo bundles seven GGUF quantizations of bowmanslayer/Qwen3.5-9B-Uncensored, for llama.cpp / LM Studio / Ollama / Jan / KoboldCpp. File sizes range from ~4.5 GB (Q3_K_M) to 18 GB (BF16) — pick one row from the table below for your device. Sister repos: full-precision bf16 in the main repo, vLLM-optimized W4A16 in -W4A16.

本仓打包 7 档 GGUF 量化,源自 bowmanslayer/Qwen3.5-9B-Uncensored, 适用于 llama.cpp / LM Studio / Ollama / Jan / KoboldCpp。文件从 ~4.5 GB(Q3_K_M) 到 18 GB(BF16),按下表挑一档匹配你的设备即可。姐妹仓:主仓是全精度 bf16, vLLM 用的 W4A16 见 -W4A16。

Vision note: this repo is text-only. The base model has a vision tower, but llama.cpp GGUF export of Qwen3.5 vision (mmproj-*.gguf) is not included here. For image understanding, use the bf16 main repo or the W4A16 repo with vLLM.

视觉说明:本仓仅文字。基座模型带视觉塔,但本仓未包 mmproj-*.gguf。要用 视觉能力请用主仓 bf16 或 W4A16 版本 + vLLM。

---

1. Which file should I download? / 该下哪一档?

Add ~1–2 GB on top of the weight size for KV cache (long context) and system overhead. Qwen3.5-9B is a hybrid-attention architecture (24 linear + 8 full-attention layers, hidden 3584), so KV cache is unusually small compared to same-sized dense models — long contexts are cheap here.

选档时在权重大小上加 1–2 GB 给 KV cache(长上下文)与 system overhead。Qwen3.5-9B 是混合注意力架构(24 层线性 + 8 层 full,hidden 3584),KV cache 比同规模稠密模型小得多, 长上下文很便宜。

Rules of thumb / 经验法则:

• First-time user, don't know what to pick → Q5_K_M (works on almost any 8+ GB GPU or Mac 16 GB, output quality is hard to distinguish from BF16 for normal use) 第一次用不知道选什么 → Q5_K_M(几乎任何 8 GB+ GPU 或 Mac 16 GB 都能跑,日常使用 与 BF16 差异极难察觉)
• Highest quality that fits → Q6_K on 12 GB, Q8_0 on 16 GB, BF16 on 24 GB 在显存范围内追求最高质量 → 12 GB 用 Q6_K,16 GB 用 Q8_0,24 GB 用 BF16
• Squeeze into 6-8 GB → IQ4_XS (smallest 4-bit file, uses IQ dequant so slower per token but fits smaller cards well) 要塞进 6-8 GB → IQ4_XS(最小 4-bit 文件,IQ 反量化更耗算力但装得下小卡)
• Must run on 6 GB or on Mac 8 GB → Q3_K_M (quality is noticeably lower — only use it if the larger quants do not fit) 只有 6 GB / Mac 8 GB → Q3_K_M(质量明显更低 —— 更大档装不下才用它)
• Apple Silicon Mac: Metal backend runs GGUF natively. 8 GB Macs → Q4_K_M / IQ4_XS / Q3_K_M; 16 GB → Q6_K or Q5_K_M; 24 GB+ → Q8_0 or BF16. Apple Silicon Mac:Metal 后端原生跑 GGUF。8 GB Mac → Q4_K_M / IQ4_XS / Q3_K_M;16 GB → Q6_K 或 Q5_K_M;24 GB+ → Q8_0 或 BF16

---

2. Full performance evaluation / 完整性能测试

All numbers below are measured on the bf16 reference build (from which every GGUF quant here was converted). Any per-quant deviation is expected to be small and in-line with llama.cpp community measurements for the same quant level.

以下数字取自 bf16 参考版(本仓每档 GGUF 都由它转换而来)。各档相对 bf16 的偏差预期较小, 与 llama.cpp 社区对同档量化的一般观察一致。

2.1 Refusal rate / 拒绝率

100 harmful prompts from mlabonne/harmful_behaviors test split, 50 harmless controls. Greedy decoding, thinking off. Fixed judge model reads full response text.

100 条 harmful 来自 mlabonne 数据集 test split,50 条 harmless 对照。贪心解码,思考关闭。 固定裁判读完整回复。

2.2 11-benchmark capability comparison / 11 项能力对比

Same fixed seed, same harness, same items. All 4 models run under vLLM TP=2 + fp8_e5m2 KV + 24576 max output tokens, thinking on.

同一固定 seed,同 harness,同题,4 模型都在 vLLM TP=2 + fp8_e5m2 KV + 24576 max output tokens 下跑,thinking on。

Among the abliterated family (excluding base): 7 firsts out of 11, mean 76.14 leads Heretic-v2 (+3.5pp) and Huihui (+17.6pp).

在 abliterated 家族中(不算 base):7 项第一 / 11,均值 76.14 领先 Heretic-v2 3.5pp, 碾压 Huihui 17.6pp。

† TruthfulQA: serial-run contamination pulled it to 54, single-run isolation gives 62.68. Table uses the fair number. Behavioral drift not knowledge loss (see main repo §3).

† TruthfulQA:串行运行 vLLM 状态污染拉低到 54,单跑 62.68。表内取单跑。这是行为漂移 不是知识损伤(详见主仓 §3)。

2.3 Output brevity / 输出简洁度

This model is the only ablation more concise than base — median -17.7%, truncation -15%. Community ablations expand across the board (Heretic-v2 +66%, Huihui nearly 2×).

本模型是唯一比 base 更简洁的消融(中位 -17.7%,截断 -15%);社区版全线膨胀(Heretic-v2 +66%,Huihui 差点翻倍)。

---

3. llama.cpp usage / llama.cpp 使用

Server mode (recommended, int8 KV cache for larger context): 服务器模式(推荐,开 int8 KV 拿到更大上下文):

./llama-server \
  -m Qwen3.5-9B-Uncensored-Q6_K.gguf \
  --ctx-size 32768 \
  --n-gpu-layers -1 \
  -ctk q8_0 -ctv q8_0 \
  --host 0.0.0.0 --port 8080

Single-shot inference / 单条推理:

./llama-cli -m Qwen3.5-9B-Uncensored-Q6_K.gguf -p "Hello" \
  -n 512 --n-gpu-layers -1 -ctk q8_0 -ctv q8_0

Note on -ctk q8_0 -ctv q8_0: default llama.cpp uses fp16 KV cache. Adding int8 KV cuts KV memory ~50% for a negligible accuracy hit — most community launch scripts omit this and pay 2× the KV memory unnecessarily.

关于 -ctk q8_0 -ctv q8_0:llama.cpp 默认 fp16 KV cache;加 int8 KV 减半 KV 内存, 精度损失可忽略。大部分社区帖不给,默认体验只到我们一半。

---

4. When to pick GGUF vs W4A16 / GGUF 还是 W4A16?

• Single user, laptop, Mac, LM Studio / Ollama / Jan / KoboldCpp → GGUF here (Q5_K_M or Q6_K are the sweet spots) 单用户、笔记本、Mac、桌面客户端 → GGUF 本仓(Q5_K_M 或 Q6_K 甜点档)

• Server with concurrent requests, high throughput, or very long context → W4A16 + vLLM (-W4A16) 服务器多并发 / 高吞吐 / 大上下文 → W4A16 + vLLM(-W4A16 姐妹仓)

• Multi-modal (image) input → main repo bf16 + vLLM (this GGUF repo is text-only) 多模态(图像)输入 → 主仓 bf16 + vLLM(本 GGUF 仓仅文字)

---

① Provenance and attribution / 来源与归属

Not an official Qwen release; not endorsed by or affiliated with the Qwen team or Alibaba Cloud. "Qwen3.5" identifies the upstream model this artifact derives from (Apache-2.0 §6).

非 Qwen 官方发布;与 Qwen 团队及阿里云无背书或从属关系。"Qwen3.5" 仅用于标识本产物所派生 的上游模型(Apache-2.0 §6)。

---

② Safety alignment has been removed / 安全对齐已被移除

This is the point of the model, and you should read this before downloading. 这是模型的核心属性,下载前请务必阅读。

The upstream ablation removes the refusal behaviour trained into Qwen/Qwen3.5-9B. Automated refusal testing on the bf16 reference recorded 2/100 on 100 held-out adversarial prompts, versus 99/100 for the base. Quantization does not restore refusals.

上游消融移除了 Qwen/Qwen3.5-9B 训练进去的拒绝行为。bf16 参考版在 100 条 held-out 对抗 提示上自动测试记录到 2/100,而基座是 99/100。量化不恢复拒绝。

Consequences you are accepting / 你正在接受的后果:

• It will produce content the original model declines to produce, including content that is offensive, dangerous, or illegal in your jurisdiction. 会产生原模型拒绝产生的内容,包括在你所在司法辖区内可能属于冒犯性、危险或违法的内容
• It has no content filter. There is no safe-completion path, no refusal fallback, no guardrail to fail back to. 没有内容过滤;没有 safe-completion 路径,没有拒绝回落,没有护栏可依
• Refusal removal is not proven exhaustive — absence of observed refusals in testing is not proof that none remain, and equally not proof that no harmful behaviour was introduced. 拒绝移除未证明穷尽
• TruthfulQA drops 24 pp vs base — behavioral, not knowledge loss (see main repo). TruthfulQA 相对基座掉 24 pp —— 行为性,非知识损伤(见主仓)

Not intended for / 不适用于:

• Deployment to third parties without your own safety layer / 未搭建自己的安全层就部署给 第三方
• Anyone under the legal age in their jurisdiction / 未达所在司法辖区法定年龄的人
• Any use prohibited by Qwen's acceptable use policy / Qwen 可接受使用政策禁止的任何用途

Intended for / 适用于: local inference and research, by people who understand the above and take responsibility for it / 本地推理与研究,由理解以上内容并为其负责的用户使用。

---

③ No warranty; responsibility rests with the user / 无担保;责任在用户

This model is provided "AS IS", without warranty of any kind, express or implied, including but not limited to warranties of merchantability, fitness for a particular purpose, and non-infringement.

本模型按**"现状"提供,不提供任何明示或默示的担保**,包括但不限于对适销性、特定用途 适用性以及无侵权性的担保。

• I do not endorse, recommend, or condone any particular use of this model. 我不背书、不推荐、不认可对本模型的任何特定使用方式
• I make no representation that its outputs are accurate, lawful, or fit for any purpose. 我不对其输出的准确性、合法性或用途适用性作任何声明
• You are solely responsible for what you generate with it, for how you deploy it, and for compliance with all laws and regulations applicable to you — including but not limited to laws on illegal content, data protection, export control, and AI-specific regulation in your jurisdiction. 你独自承担全部责任:对生成内容、部署方式、以及遵守适用于你的一切法律法规负责
• To the maximum extent permitted by law, I accept no liability for any claim, damage, or other liability arising from the model or its use. 在法律允许的最大范围内,我不承担因本模型或其使用而产生的任何索赔、损害或其他责任

Downloading these files means you accept the above. If you do not, do not download them. 下载本仓文件即表示你接受以上内容。若不接受,请勿下载。

The author is not a lawyer; the above is not legal advice. 作者不是律师;以上不构成法律意见。

---

License

Apache 2.0, inherited through the chain above. The Apache-2.0 grant covers the weights; it does not grant permission for uses that are unlawful where you are.

Apache 2.0,依上述归属链继承。Apache-2.0 授权覆盖权重本身;不授权任何在你所在地不合法 的用途。