Downloads · 30 days
2.2K
52% of all-time downloads
Yirasumi/Huihui-Qwen3.8-27B-abliterated-INT4-W4A16
Huihui-Qwen3.8-27B-abliterated-INT4-W4A16 is a machine learning model from Yirasumi. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
INT4 W4A16 quantization of huihui-ai/Huihui-Qwen3.8-27B-abliterated, produced with Intel AutoRound and exported in compressed-tensors (pack-quantized) format.
Downloads · 30 days
2.2K
52% of all-time downloads
All-time downloads
4.3K
Public
Parameters
5.2B
17.6 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors17.6 GB · 100%
How the weights are stored.
I3225.6B · 92%
From the Hugging Face model README
INT4 W4A16 quantization of huihui-ai/Huihui-Qwen3.8-27B-abliterated, produced with Intel AutoRound and exported in compressed-tensors (pack-quantized) format.
| Property | Value |
|---|---|
| Base model | huihui-ai/Huihui-Qwen3.8-27B-abliterated |
| Quantization | INT4 weight-only (W4A16), group_size=128, symmetric |
| Algorithm | AutoRound (Intel) |
| Format | compressed-tensors / pack-quantized (auto-detected by vLLM) |
| lm_head | Quantized to INT4 |
| Weight size | ~15.6 GiB (4 shards) |
| Architecture | Qwen3_5ForConditionalGeneration (multimodal, vision tower kept BF16) |
| Context | 262,144 native |
| License | Apache-2.0 |
This checkpoint is specifically sized to run fast on a single 24 GB GPU (e.g. RTX 3090):
lm_head to INT4 frees ~1.9 GB of VRAM compared to BF16-lm_head quants.The language-model transformer layers are quantized; the vision tower, linear_attn.in_proj_a/b and the MTP fusion layer (mtp.fc) remain in BF16.
vllm serve Yirasumi/Huihui-Qwen3.8-27B-abliterated-INT4-W4A16 \
--served-model-name qwen3.8-27b \
--max-model-len 40000 \
--gpu-memory-utilization 0.93 \
--kv-cache-dtype fp8_e5m2 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice --tool-call-parser qwen3_xml \
--enable-prefix-caching --enable-chunked-prefill
MTP speculative decoding (the checkpoint ships the Qwen3.5 MTP head in model_extra_tensors.safetensors):
vllm serve Yirasumi/Huihui-Qwen3.8-27B-abliterated-INT4-W4A16 \
--speculative-config '{"method":"mtp","num_speculative_tokens":4}' \
# ... same flags as above
On a 24 GB RTX 3090 the tuned serving stack from syv-ai/qwen38-27b-rtx3090 applies directly — see that repo's prepare/ scripts for the optional lm_head/embed requantization and draft-vocab steps.
auto_round:llm_compressor export)model.safetensors.index.json + 4 shards + model_extra_tensors.safetensors (MTP head) all presentcompressed-tensors path (Marlin INT4 kernels)quant_method: compressed-tensors, format: pack-quantizedUsage warning: This is an abliterated model. Safety filtering has been significantly reduced. Use responsibly and in compliance with applicable laws.