Downloads · 30 days
87
21% of all-time downloads
abcdsystems/Qwen3.5-9B-Abliterated-GDN-Hybrid-EXL3-6bpw
Qwen3.5-9B-Abliterated-GDN-Hybrid-EXL3-6bpw is a text generation model from abcdsystems. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
EXL3 (exllamav3) quantization of huihui-ai/Huihui-Qwen3.5-9B-abliterated at 6.0 bits per weight.
Downloads · 30 days
87
21% of all-time downloads
All-time downloads
418
Public
Parameters
4.5B
9 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors9 GB · 100%
How the weights are stored.
I163B · 67%
From the Hugging Face model README
EXL3 (exllamav3) quantization of huihui-ai/Huihui-Qwen3.5-9B-abliterated at 6.0 bits per weight.
Quantized by ABC&D System Inc for production real-time inference on a single RTX 3090.
| Property | Value |
|---|---|
| Base model | Qwen/Qwen3.5-9B |
| Abliteration | huihui-ai via remove-refusals-with-transformers |
| Parameters | 9B dense (all active per token) |
| Architecture | Qwen3_5ForConditionalGeneration — GatedDeltaNet + Attention hybrid |
| Quantization | EXL3 6.0bpw, head bits 6, codebook mul1 |
| Calibration | 250 rows × 2048 cols |
| VRAM loaded | ~5.6GB model + KV cache |
| Total size on disk | ~8.5GB |
| License | Apache 2.0 (Alibaba Cloud) |
Qwen3.5 is not a standard transformer. It uses a hybrid attention design that alternates between two fundamentally different attention mechanisms:
[linear, linear, linear, FULL, linear, linear, linear, FULL, ...] × 4 = 32 layers
24 GatedDeltaNet layers + 8 full attention layers. The linear layers handle local/sequential patterns efficiently while the full attention layers provide global context anchoring.
The model includes a 1-layer MTP head for predicting multiple next tokens simultaneously, enabling faster inference throughput than standard single-token autoregressive decoding.
Despite the architecture string Qwen3_5ForConditionalGeneration (which Alibaba uses for all Qwen3.5 variants including VL), this model contains zero vision weights. No ViT encoder, no image projector. The architecture class name reflects the MTP head and hybrid design, not multimodal capability.
Quantized with exllamav3 v1.1.0.
# config.yml
network:
host: 0.0.0.0
port: 11435
model:
model_dir: /path/to/models
model_name: Qwen3.5-9B-Abliterated-GDN-Hybrid-EXL3-6bpw
max_seq_len: 4096
gpu_split: [24]
cache_mode: FP16
CUDA_VISIBLE_DEVICES=0 python3 main.py --config config.yml
Qwen3.5 has a built-in chain-of-thought reasoning mode that is on by default. For real-time applications (voice, chat, API), you must disable it or the model will consume your entire token budget on internal reasoning before producing visible output.
Add to your system prompt:
/no_think
Or set enable_thinking: false in your chat template parameters.
The raw exllamav3 Generator Python API may fail to initialize GatedDeltaNet recurrent state (conv_state on meta device error). TabbyAPI handles this initialization correctly. Use TabbyAPI as the serving layer rather than calling exllamav3 directly.
| Config | VRAM | Notes |
|---|---|---|
| Minimum | ~8GB | Model only, minimal KV cache |
| Recommended | 16GB+ | Model + FP16 KV cache for 4096 context |
| Tested on | RTX 3090 24GB | 5.6GB model, 18.4GB free for cache |
This quantization was built for low-latency, single-user dedicated inference — specifically real-time voice AI over SIP/phone calls. The abliteration ensures zero self-censorship in commercial conversation contexts.
Good for:
Not ideal for:
This model has had its safety filtering surgically removed (abliterated). It will comply with any instruction without refusal. Users are solely responsible for ensuring their use complies with applicable laws and ethical standards. See huihui-ai's original warnings for full details.