Downloads · 30 days
166
24% of all-time downloads
zeallat/Huihui-Qwen3.8-27B-abliterated-AWQ-MTP
Huihui-Qwen3.8-27B-abliterated-AWQ-MTP is a text generation model from zeallat. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
4-bit weight-only AWQ quantization of huihui-ai/Huihui-Qwen3.8-27B-abliterated, produced with llm-compressor 0.13.0. The Multi-Token Prediction (MTP) head is kept in bf16 and shipped with the checkpoint so
Downloads · 30 days
166
24% of all-time downloads
All-time downloads
688
Public
Parameters
27.8B
27.8 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors27.7 GB · 100%
How the weights are stored.
I3218.8B · 68%
From the Hugging Face model README
4-bit weight-only AWQ quantization of
huihui-ai/Huihui-Qwen3.8-27B-abliterated,
produced with llm-compressor 0.13.0.
The Multi-Token Prediction (MTP) head is kept in bf16 and shipped with the checkpoint so
vLLM speculative decoding works out of the box.
W4A16_ASYM, group size 128, compressed-tensors pack-quantized formatLinear modules in the full-attention layers and all MLPslm_head, the vision tower (visual.*), the MTP head (mtp.*),
and every GatedDeltaNet linear-attention projection (linear_attn.*)HuggingFaceH4/ultrachat_200k, max_seq_length=1024The model is a hybrid: of its 64 text layers, 48 are GatedDeltaNet (linear_attn) and 16
are full attention (full_attention_interval=4). AWQ activation smoothing has no
well-formed mapping for the GatedDeltaNet blocks, so those projections are excluded from
both smoothing and quantization rather than approximated. v_proj -> o_proj smoothing is
also skipped on the full-attention layers, whose GQA + output-gate shapes do not line up.
vllm serve zeallat/Huihui-Qwen3.8-27B-abliterated-AWQ-MTP \
--speculative-config '{"method":"mtp","num_speculative_tokens":1}'
The log should report a detected MTP model. Dropping --speculative-config runs the model
without speculative decoding.
save_pretrained does not carry the MTP head through quantization, so the 15 mtp.*
tensors were copied back from the source checkpoint in bf16, the
model.safetensors.index.json weight map was regenerated to cover them, and
re:.*mtp.* is present in quantization_config.ignore. Without all three, vLLM either
tries to build a quantized drafter and fails, or silently loads an untrained draft head
(0% acceptance).
If you previously served this model and then changed quantization_config, clear the
torch.compile cache before rebooting.
This is an abliterated (refusal-behaviour-removed) base model. Safety alignment has been weakened upstream; you are responsible for how you deploy it.