Mistral-Nemo-Instruct-2407-abliterated-AWQ

Mistral Nemo Instruct 2407 Abliterated — AWQ (W4A16)

AWQ 4-bit quantization of natong19/Mistral-Nemo-Instruct-2407-abliterated, itself an abliteration of Mistral Nemo Instruct 2407.

Why this exists

The base abliterated model was only published in BF16 and GGUF. This AWQ quant is ~7 GB VRAM and serves natively in vLLM with the Marlin kernel for fast inference.

Quantization details

• Tool: llm-compressor (vLLM project)
• Format: compressed-tensors (auto-detected by vLLM)
• Scheme: W4A16_ASYM (4-bit asymmetric weights, group_size 128, 16-bit activations)
• Algorithm: AWQ (Activation-aware Weight Quantization)
• Calibration: 128 French samples × 192 tokens from wikimedia/wikipedia (20231101.fr)
• lm_head left in fp16 (preserves output quality)
• Note: v_proj → o_proj smoothing is skipped on all 40 layers due to GQA shape mismatch (32 query heads / 8 KV heads). This is standard AWQ behaviour for GQA models — quality impact is marginal since the input_layernorm → q/k/v smoothing remains active.

Usage with vLLM

vllm serve KebalBaguette/Mistral-Nemo-Instruct-2407-abliterated-AWQ \
  --quantization compressed-tensors \
  --max-model-len 8192

License

Apache 2.0, inherited from Mistral Nemo and the abliterated base.