Downloads · 30 days
14
22% of all-time downloads
Phuc2005/Qwen2.5-7B-AWQ-W4A16-g128
Qwen2.5-7B-AWQ-W4A16-g128 is a text generation model from Phuc2005. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
AWQ-quantized base model derived from Qwen/Qwen2.5-7B. Weights are W4A16 (4-bit weights, FP16 activations), group size 128, zero point enabled, and use the original MIT Han Lab llm-awq GEMM layout. This is a base mode…
Downloads · 30 days
14
22% of all-time downloads
All-time downloads
64
Public
Parameters
14.2B
5.7 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors5.7 GB · 100%
How the weights are stored.
I1613.1B · 92%
From the Hugging Face model README
AWQ-quantized base model derived from Qwen/Qwen2.5-7B.
Weights are W4A16 (4-bit weights, FP16 activations), group size 128, zero point enabled,
and use the original MIT Han Lab llm-awq GEMM layout. This is a base model, not an
instruction-tuned/chat model.
An NVIDIA CUDA GPU is required by this llm-awq backend. Reproduce the pinned environment:
git clone https://github.com/mit-han-lab/llm-awq.git
cd llm-awq
git checkout d6e797a42b9ef7778de8ee2352116e0f48a78d61
pip install -e .
cd awq/kernels
python setup.py install
The pinned project installs PyTorch 2.3.0, Transformers 4.46.0, and Accelerate 0.34.2. Then load directly from the Hub:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "Phuc2005/Qwen2.5-7B-AWQ-W4A16-g128"
tokenizer = AutoTokenizer.from_pretrained(repo_id, use_fast=False)
model = AutoModelForCausalLM.from_pretrained(
repo_id,
torch_dtype=torch.float16,
device_map="auto",
trust_remote_code=True,
)
prompt = "The purpose of quantization in neural networks is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))
d149729398750b98c0af14eb82c78cfe92750796d6e797a42b9ef7778de8ee2352116e0f48a78d617.139332294464111quantization_metadata.json contains the full environment and packaging record. The
WikiText score validates the quantized checkpoint; it does not prove equal quality on every
dataset or downstream task.
model-00001-of-00002.safetensors: 4fb746ceebc137381e8fdb89e14bb5c226e02bf20dd24007f51cd35fc83ab34cmodel-00002-of-00002.safetensors: 440eb69d3cdeb5e321679c1c466d6c46878ff0f5c69cdc71cc7d5418ba0c43e4source/qwen2.5-7b-w4-g128-awq.pt: 4a479b93697fa5ffc444db59530a6896b60ff99430cb82ce24f040499571f3b5This repository is distributed under Apache-2.0; see LICENSE.