Downloads · 30 days
8
3% of all-time downloads
yujiepan/Meta-Llama-3-8B-gptq-w4g64
Meta-Llama-3-8B-gptq-w4g64 is a text generation model from yujiepan. Use it when you need the model to write or continue text. It is set up for transformers.
Note: this repo has low accuracy and is under investigation.
Downloads · 30 days
8
3% of all-time downloads
All-time downloads
297
Public
Parameters
8B
5.9 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors5.9 GB · 100%
How the weights are stored.
I327B · 87%
From the Hugging Face model README
Note: this repo has low accuracy and is under investigation.
This model applies AutoGPTQ on meta-llama/Meta-Llama-3-8B.
| model | precision | wikitext ppl (↓) |
|---|---|---|
| meta-llama/Meta-Llama-3-8B | FP16 | 9.179 |
| yujiepan/Meta-Llama-3-8B-gptq-w4g64 | w4g64 | 10.097 |
Note:
import os
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig
model_id = "meta-llama/Meta-Llama-3-8B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
quantization_config = GPTQConfig(
bits=4, group_size=64,
dataset="c4-new",
tokenizer=tokenizer,
)
model = AutoModelForCausalLM.from_pretrained(
model_id,
device_map="auto",
low_cpu_mem_usage=True,
quantization_config=quantization_config,
)
model.push_to_hub('yujiepan/Meta-Llama-3-8B-gptq-w4g64')