Downloads · 30 days
17
5% of all-time downloads
yujiepan/Meta-Llama-3-8B-awq-w4g64
Meta-Llama-3-8B-awq-w4g64 is a text generation model from yujiepan. Use it when you need the model to write or continue text. It is set up for transformers.
This model applies AutoAWQ on meta-llama/Meta-Llama-3-8B.
Downloads · 30 days
17
5% of all-time downloads
All-time downloads
368
Public
Parameters
8B
5.9 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors5.9 GB · 100%
How the weights are stored.
I327B · 87%
From the Hugging Face model README
This model applies AutoAWQ on meta-llama/Meta-Llama-3-8B.
| model | precision | wikitext ppl (↓) |
|---|---|---|
| meta-llama/Meta-Llama-3-8B | FP16 | 9.179 |
| yujiepan/Meta-Llama-3-8B-awq-w4g64 | w4g64 | 9.219 |
Note:
from awq import AutoAWQForCausalLM
model = AutoAWQForCausalLM.from_quantized('yujiepan/Meta-Llama-3-8B-awq-w4g64')
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer
model_path = "meta-llama/Meta-Llama-3-8B"
quant_config = {"zero_point": True, "q_group_size": 64, "w_bit": 4, "version": "GEMM"}
model = AutoAWQForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model.quantize(tokenizer, quant_config=quant_config)