Llama-3-8B-Instruct-AWQ-W4A16-g128

Llama-3-8B-Instruct-AWQ-W4A16-g128

AWQ-quantized model derived from meta-llama/Meta-Llama-3-8B-Instruct. Weights are W4A16 (4-bit weights, FP16 activations), group size 128, zero point enabled, and use the original MIT Han Lab llm-awq GEMM layout. This is the instruction-tuned/chat model. The original tokenizer chat template is included and should be used for inference.

Load and run

An NVIDIA CUDA GPU is required by this llm-awq backend. Reproduce the pinned environment:

git clone https://github.com/mit-han-lab/llm-awq.git
cd llm-awq
git checkout d6e797a42b9ef7778de8ee2352116e0f48a78d61
pip install -e .
cd awq/kernels
python setup.py install

The pinned project installs PyTorch 2.3.0, Transformers 4.46.0, and Accelerate 0.34.2. Then load directly from the Hub:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "Phuc2005/Llama-3-8B-Instruct-AWQ-W4A16-g128"
tokenizer = AutoTokenizer.from_pretrained(repo_id, use_fast=False)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    torch_dtype=torch.float16,
    device_map="auto",
    trust_remote_code=True,
)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain neural-network quantization briefly."},
]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
).to(model.device)
output = model.generate(**inputs, max_new_tokens=64, do_sample=False)
new_tokens = output[0, inputs["input_ids"].shape[1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))

Quantization and validation

• Source revision: 8afb486c1db24fe5011ec46dfbe5b5dccdb575c2
• AWQ implementation commit: d6e797a42b9ef7778de8ee2352116e0f48a78d61
• Quantization: real W4A16, 4 bits, group size 128
• Official llm-awq WikiText perplexity: 8.572953224182129

quantization_metadata.json contains the full environment and packaging record. The WikiText score validates the quantized checkpoint; it does not prove equal quality on every dataset or downstream task.

SHA-256

• model-00001-of-00002.safetensors: 4356fc775d3613e21f2190ca26d51ebfefb173bcc7a3295f0fef151a89cc9d80
• model-00002-of-00002.safetensors: 1e40c69881fda0e64cd76a5a2ce143b3bbba53b2c7cbce72cc500af0a1bd037c
• source/llama3-instruct-8b-w4-g128-awq.pt: 837f6899a065096bbca1094bb9f6df8f3d4f787d41c2ab9f1324a05ea75e2a0a

License

This derivative is distributed under the Meta Llama 3 Community License. See LICENSE, NOTICE, and USE_POLICY.md. Built with Meta Llama 3.