Downloads · 30 days
25
3% of all-time downloads
j-a-a-a-y/Huihui-Qwen3.5-27B-abliterated-W4A16-compressed-tensors
Huihui-Qwen3.5-27B-abliterated-W4A16-compressed-tensors is a machine learning model from j-a-a-a-y. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as other.
4-bit weight quantization of huihui-ai/Huihui-Qwen3.5-27B-abliterated using llm-compressor W4A16 scheme.
Downloads · 30 days
25
3% of all-time downloads
All-time downloads
746
Public
Parameters
27.1B
17.7 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors17.6 GB · 100%
How the weights are stored.
I3224.4B · 90%
From the Hugging Face model README
4-bit weight quantization of huihui-ai/Huihui-Qwen3.5-27B-abliterated using llm-compressor W4A16 scheme.
| Property | Value |
|---|---|
| Base model | Qwen3.5-27B (abliterated) |
| Quantization | W4A16 (4-bit weights, 16-bit activations) |
| Format | compressed-tensors (vLLM native) |
| Size on disk | 17.6 GB |
| GPU VRAM | ~16 GB (fits RTX 5090 32GB with MTP + KV cache) |
| Calibration | 128 samples from Pile validation set |
python -m vllm.entrypoints.openai.api_server \
--model j-a-a-a-y/Huihui-Qwen3.5-27B-abliterated-W4A16-compressed-tensors \
--served-model-name qwen3.5-27b \
--dtype float16 \
--max-model-len 4096 \
--speculative-config '{"method": "mtp", "num_speculative_tokens": 5}' \
--performance-mode interactivity
Benchmarked with MTP=5 speculative decoding + CUDA graphs + interactivity mode:
| Metric | Value |
|---|---|
| Single request (256 tok) | ~149 tok/s |
| Single request (512 tok) | ~131 tok/s |
| Batch=4 aggregate | ~410 tok/s |
| MTP acceptance rate | 50% |
Quantized with vLLM's llm-compressor using the W4A16 scheme:
Compared to GPTQ W4A16: ~2 GB smaller on disk, ~1.5 GB less VRAM, same inference speed (both use Marlin kernel).