Downloads · 30 days
112
3% of all-time downloads
j-a-a-a-y/Huihui-Qwen3.5-27B-abliterated-AWQ-W4A16
Huihui-Qwen3.5-27B-abliterated-AWQ-W4A16 is a machine learning model from j-a-a-a-y. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as other.
AWQ 4-bit quantization of huihui-ai/Huihui-Qwen3.5-27B-abliterated using AutoAWQ with Qwen3.5 support patches.
Downloads · 30 days
112
3% of all-time downloads
All-time downloads
4K
Public
Parameters
27.3B
18.6 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors18.6 GB · 100%
How the weights are stored.
I3224.3B · 89%
From the Hugging Face model README
AWQ 4-bit quantization of huihui-ai/Huihui-Qwen3.5-27B-abliterated using AutoAWQ with Qwen3.5 support patches.
Text-only (vision encoder removed). MTP head preserved for speculative decoding.
| Property | Value |
|---|---|
| Quantization | AWQ W4A16 (group_size=128, zero_point=True) |
| Size on disk | 18.6 GB |
| MTP head | Included (BF16, 0.85 GB) |
| Vision encoder | Removed (-0.92 GB) |
| Calibration | 128 samples, Pile validation |
When used with MTP speculative decoding, this AWQ quantization shows lower MTP acceptance (31%) compared to GPTQ W4A16 (50%). This results in ~48% lower single-request throughput. For MTP-enabled serving, GPTQ is recommended. See j-a-a-a-y/Huihui-Qwen3.5-27B-abliterated-GPTQ-W4A16.
| Metric | GPTQ W4A16 | This AWQ |
|---|---|---|
| Single 256 tok | 148.8 tok/s | 76.7 tok/s |
| MTP acceptance | 50% | 31% |
| Batch=4 agg | 410 tok/s | 313 tok/s |
python -m vllm.entrypoints.openai.api_server \
--model j-a-a-a-y/Huihui-Qwen3.5-27B-abliterated-AWQ-W4A16 \
--served-model-name qwen3.5-27b \
--dtype float16 \
--quantization awq_marlin \
--speculative-config '{"method": "mtp", "num_speculative_tokens": 5}'