Qwen2.5-7B-Instruct-AWQ-W4A16-g128

Qwen2.5-7B-Instruct-AWQ-W4A16-g128

AWQ-quantized model derived from Qwen/Qwen2.5-7B-Instruct. Weights are W4A16 (4-bit weights, FP16 activations), group size 128, zero point enabled, and use the original MIT Han Lab llm-awq GEMM layout. This is the instruction-tuned/chat model. The original tokenizer chat template is included and should be used for inference.

Load and run

An NVIDIA CUDA GPU is required by this llm-awq backend. Reproduce the pinned environment:

git clone https://github.com/mit-han-lab/llm-awq.git
cd llm-awq
git checkout d6e797a42b9ef7778de8ee2352116e0f48a78d61
pip install -e .
cd awq/kernels
python setup.py install

The pinned project installs PyTorch 2.3.0, Transformers 4.46.0, and Accelerate 0.34.2. Then load directly from the Hub:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "Phuc2005/Qwen2.5-7B-Instruct-AWQ-W4A16-g128"
tokenizer = AutoTokenizer.from_pretrained(repo_id, use_fast=False)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    torch_dtype=torch.float16,
    device_map="auto",
    trust_remote_code=True,
)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain neural-network quantization briefly."},
]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
).to(model.device)
output = model.generate(**inputs, max_new_tokens=64, do_sample=False)
new_tokens = output[0, inputs["input_ids"].shape[1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))

Quantization and validation

• Source revision: a09a35458c702b33eeacc393d103063234e8bc28
• AWQ implementation commit: d6e797a42b9ef7778de8ee2352116e0f48a78d61
• Quantization: real W4A16, 4 bits, group size 128
• Official llm-awq WikiText perplexity: 7.731052398681641

quantization_metadata.json contains the full environment and packaging record. The WikiText score validates the quantized checkpoint; it does not prove equal quality on every dataset or downstream task.

SHA-256

• model-00001-of-00002.safetensors: 92df65cc9ee546a8bb7d1d10e2b6d88df49888f2a04a8378462b14bcdf2b540e
• model-00002-of-00002.safetensors: 21b864adc392aee9e6f2f3e46fe1cd271f5d1548bf81d190a2240c74c191bfa5
• source/qwen2.5-7b-instruct-w4-g128-awq.pt: 46cdee9643486db34cd63bd466af7394df8b0895b269177305ecfb1383ce2e1b

License

This repository is distributed under Apache-2.0; see LICENSE.