Downloads · 30 days
366
7% of all-time downloads
KebalBaguette/Qwen2.5-7B-Instruct-abliterated-v2-AWQ
Qwen2.5-7B-Instruct-abliterated-v2-AWQ is a machine learning model from KebalBaguette. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
AWQ 4-bit quantization of huihui-ai/Qwen2.5-7B-Instruct-abliterated-v2, itself an abliteration of Qwen2.5 7B Instruct.
Downloads · 30 days
366
7% of all-time downloads
All-time downloads
5.4K
Public
Parameters
2B
5.6 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors5.6 GB · 100%
How the weights are stored.
I326.6B · 85%
From the Hugging Face model README
AWQ 4-bit quantization of huihui-ai/Qwen2.5-7B-Instruct-abliterated-v2,
itself an abliteration of Qwen2.5 7B Instruct.
The base abliterated model was only published in BF16 and GGUF. This AWQ quant serves natively in vLLM with the Marlin kernel for fast inference.
wikimedia/wikipedia (20231101.fr)lm_head left in fp16 (preserves output quality)v_proj → o_proj smoothing is skipped on all 28 layers due to GQA shape mismatch (28 query heads / 4 KV heads). Standard AWQ behaviour for GQA models — quality impact is marginal since the input_layernorm → q/k/v smoothing remains active.vllm serve KebalBaguette/Qwen2.5-7B-Instruct-abliterated-v2-AWQ \
--quantization compressed-tensors \
--tool-call-parser hermes \
--enable-auto-tool-choice \
--max-model-len 8192
Apache 2.0, inherited from the upstream model and the abliterated base.