Downloads · 30 days
6
10% of all-time downloads
Phuc2005/Llama-3-8B-AWQ-W4A16-g128
Llama-3-8B-AWQ-W4A16-g128 is a text generation model from Phuc2005. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as llama3.
AWQ-quantized base model derived from meta-llama/Meta-Llama-3-8B. Weights are W4A16 (4-bit weights, FP16 activations), group size 128, zero point enabled, and use the original MIT Han Lab llm-awq GEMM layout. This is…
Downloads · 30 days
6
10% of all-time downloads
All-time downloads
60
Public
Parameters
15.1B
5.8 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors5.8 GB · 100%
How the weights are stored.
I1614B · 93%
From the Hugging Face model README
AWQ-quantized base model derived from meta-llama/Meta-Llama-3-8B.
Weights are W4A16 (4-bit weights, FP16 activations), group size 128, zero point enabled,
and use the original MIT Han Lab llm-awq GEMM layout. This is a base model, not an
instruction-tuned/chat model.
An NVIDIA CUDA GPU is required by this llm-awq backend. Reproduce the pinned environment:
git clone https://github.com/mit-han-lab/llm-awq.git
cd llm-awq
git checkout d6e797a42b9ef7778de8ee2352116e0f48a78d61
pip install -e .
cd awq/kernels
python setup.py install
The pinned project installs PyTorch 2.3.0, Transformers 4.46.0, and Accelerate 0.34.2. Then load directly from the Hub:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "Phuc2005/Llama-3-8B-AWQ-W4A16-g128"
tokenizer = AutoTokenizer.from_pretrained(repo_id, use_fast=False)
model = AutoModelForCausalLM.from_pretrained(
repo_id,
torch_dtype=torch.float16,
device_map="auto",
trust_remote_code=True,
)
prompt = "The purpose of quantization in neural networks is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))
8cde5ca8380496c9a6cc7ef3a8b46a0372a1d920d6e797a42b9ef7778de8ee2352116e0f48a78d616.546638488769531quantization_metadata.json contains the full environment and packaging record. The
WikiText score validates the quantized checkpoint; it does not prove equal quality on every
dataset or downstream task.
model-00001-of-00002.safetensors: 363c6b6b44dbec006ba5ed5c04b9f9ba7c60fdd990e4c4e5b1ab5360b7b7aeb5model-00002-of-00002.safetensors: f3ecea694f172a2cb25e8123aa8626e30ff588237119149eba99a28bc47bf86esource/llama3-8b-w4-g128-awq.pt: 6de86aecfc9eadead0966366004594704bb747c8acfde46fbf14cf16706e7617This derivative is distributed under the Meta Llama 3 Community License. See LICENSE, NOTICE, and USE_POLICY.md. Built with Meta Llama 3.