qwen2.5-1.5b-gptq-4bit-v2

Qwen2.5-1.5B-Instruct · GPTQ 4-bit (v2, quality-optimized)

Part of a systematic 4-way quantization study on Qwen2.5-1.5B-Instruct. See the study overview for comparisons across all variants.

An improved GPTQ 4-bit quantization of Qwen/Qwen2.5-1.5B-Instruct, using a quality-optimized configuration derived from ablation against GPTQ v1.

Three targeted fixes over v1:

1. Smaller group size (128 → 64): each scaling factor covers fewer weights, reducing per-group quantization error at the cost of ~5% larger file size
2. Activation ordering ON (desc_act=False → True): reorders weight columns by descending activation magnitude before quantizing, so the most influential weights are quantized last when accumulated error is lowest
3. Instruction-domain calibration: OpenHermes-2.5 chat data instead of Wikitext-2 prose — better signal for an instruction-tuned model

Result: perplexity delta reduced from +3.46 (v1) to +1.19 (v2) — a 65% improvement in quality preservation with only 5% size increase.

---

Benchmark Results

All measurements on A100-40GB, batch size 1, HuggingFace Transformers, 50 tokens generated, 10-run average.

Quality gate (PPL delta < 1.0): ❌ FAIL (borderline — 0.19 above threshold)

The marginal quality gate failure is consistent with published literature: at INT4, 1.5B models typically see +1.0–2.0 PPL degradation due to limited parameter redundancy. A delta of +1.19 represents good quality preservation for this model size and bit-width.

---

What Changed vs v1

---

Quantization Config

GPTQConfig(
    bits=4,
    group_size=64,       # finer scaling than standard 128
    desc_act=True,       # activation ordering ON — key quality improvement
    damp_percent=0.01,   # Hessian stability, default for Qwen2 architecture
    dataset=openhermes_samples,  # instruction-domain calibration
)

---

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Mohaaxa/qwen2.5-1.5b-gptq-4bit-v2",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("Mohaaxa/qwen2.5-1.5b-gptq-4bit-v2")

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain quantization in one paragraph."}
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to("cuda")

output = model.generate(**inputs, max_new_tokens=200, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Requirements: pip install transformers==4.44.0 auto-gptq==0.7.1 optimum==1.16.0

---

Why is throughput slower than v1?

desc_act=True adds a column-reordering step during quantization that produces better weights, but those reordered weights are slightly harder for the GPTQ CUDA kernel to process at inference time — hence 13.8 vs 16.1 tok/s. This is a known tradeoff. For throughput-critical deployments, see the AWQ variant which achieves 24.9 tok/s with comparable quality (+1.26 PPL).

---

Study Context

GPTQ v2 achieves the best raw perplexity of the three quantized variants (+1.19) at the cost of slowest inference (13.8 tok/s). If quality is the priority and throughput is secondary, this is the recommended variant. For deployment where both matter, AWQ provides a better overall tradeoff.

Study methodology: Mohaaxa