Downloads · 30 days
14
2% of all-time downloads
gemma-2-9b-AWQ
radi-cho/gemma-2-9b-AWQ
gemma-2-9b-AWQ is a text generation model from radi-cho. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as gemma.
AWQ-quantized package (W4G128) of google/gemma-2-9b. Support for Gemma2 in the codebase of AutoAWQ is proposed in the following pull request. To use the model, follow the AutoAWQ examples with the source from 562.
All-time downloads
669
Public
Parameters
10.2B
8 GB on disk
Likes
1
Public
Repo makeup
Click a slice to open those files.
.safetensors8 GB · 100%
Parameter types
How the weights are stored.
I328.3B · 82%
Try a prompt
Model card
From the Hugging Face model README
AWQ-quantized package (W4G128) of google/gemma-2-9b.
Support for Gemma2 in the codebase of AutoAWQ is proposed in the following pull request.
To use the model, follow the AutoAWQ examples with the source from #562.
Evaluation<br> WikiText-2 PPL: 7.08<br> C4 PPL: 11.05
Loading
model_path = "radi-cho/gemma-2-9b-AWQ"
# With transformers
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(model_path, device_map="cuda:0")
# With transformers (fused)
from transformers import AutoModelForCausalLM, AwqConfig
quantization_config = AwqConfig(bits=4, fuse_max_seq_len=512, do_fuse=True)
model = AutoModelForCausalLM.from_pretrained(model_path, quantization_config=quantization_config).to(0)
# With AutoAWQ
from awq import AutoAWQForCausalLM
model = AutoAWQForCausalLM.from_quantized(model_path)
- Task
- Text Generation
- Library
- transformers
- Type
- gemma2
- License
- gemma
- Created
- Aug 1, 2024
- Updated
- Aug 1, 2024