Qwen2.5-0.5B-Instruct-AWQ-INT4-lmhead

Qwen2.5-0.5B-Instruct - W4A16 ASYM (with Quantized lm_head)

This is a custom INT4 quantized version of the Qwen2.5-0.5B-Instruct model.

What makes this model special?

Standard AWQ models leave the lm_head in FP16 because it shares embeddings with the input layer. For this model, the lm_head was explicitly untied and quantized to W4A16 Asymmetric alongside the rest of the network.

By crushing the massive 151k-dimension vocabulary projection down to 4-bit, this model significantly reduces the memory bandwidth bottleneck during the generation phase, making it incredibly fast on hardware like the Tesla T4.

Quantization Details

• Tool: llmcompressor
• Format: compressed-tensors (Marlin compatible)
• Scheme: W4A16_ASYM (Weights: 4-bit, Activations: 16-bit)
• Calibration Data: HuggingFaceH4/ultrachat_200k (256 samples)

How to Serve with vLLM

Because this model uses the compressed-tensors format and an untied head, you must run it on vLLM v0.6.0 or higher.

vllm serve your_username/Qwen2.5-0.5B-Instruct-AWQ-INT4-lmhead \
  --max-model-len 2048 \
  --quantization compressed-tensors