Downloads · 30 days
11
14% of all-time downloads
KebalBaguette/aya-expanse-8b-abliterated-AWQ
aya-expanse-8b-abliterated-AWQ is a machine learning model from KebalBaguette. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as cc-by-nc-4.0.
AWQ 4-bit quantization of lenML/aya-expanse-8b-abliterated, itself an abliteration of Aya Expanse 8B.
Downloads · 30 days
11
14% of all-time downloads
All-time downloads
79
Public
Parameters
2B
5.7 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors5.7 GB · 100%
How the weights are stored.
I327B · 86%
From the Hugging Face model README
AWQ 4-bit quantization of lenML/aya-expanse-8b-abliterated,
itself an abliteration of Aya Expanse 8B.
The base abliterated model was only published in BF16 and GGUF. This AWQ quant serves natively in vLLM with the Marlin kernel for fast inference.
wikimedia/wikipedia (20231101.fr)lm_head left in fp16 (preserves output quality)v_proj → o_proj smoothing is skipped on all 32 layers due to GQA shape mismatch (32 query heads / 8 KV heads). Standard AWQ behaviour for GQA models — quality impact is marginal since the input_layernorm → q/k/v smoothing remains active.vllm serve KebalBaguette/aya-expanse-8b-abliterated-AWQ \
--quantization compressed-tensors \
--tool-call-parser hermes \
--enable-auto-tool-choice \
--max-model-len 8192
CC-BY-NC-4.0 — non-commercial use only. Inherited from Cohere Aya Expanse 8B and lenML's abliterated derivative. This abliterated derivative is for research, personal, and non-commercial evaluation use only. For commercial deployment, contact Cohere for licensing, or use an Apache-licensed alternative (e.g. Mistral 7B / Nemo).