Downloads · 30 days
16
10% of all-time downloads
gemma-2-2b-AWQ
radi-cho/gemma-2-2b-AWQ
gemma-2-2b-AWQ is a text generation model from radi-cho. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as gemma.
AWQ-quantized package (W4G128) of google/gemma-2-2b. Support for Gemma2 in the codebase of AutoAWQ is proposed in the following pull request. To use the model, follow the AutoAWQ examples with the source from 562.
All-time downloads
163
Public
Parameters
3.2B
3.4 GB on disk
Likes
1
Public
Repo makeup
Click a slice to open those files.
.safetensors3.4 GB · 99%
Parameter types
How the weights are stored.
I322B · 63%
Try a prompt
Model card
From the Hugging Face model README
AWQ-quantized package (W4G128) of google/gemma-2-2b.
Support for Gemma2 in the codebase of AutoAWQ is proposed in the following pull request.
To use the model, follow the AutoAWQ examples with the source from #562.
Evaluation<br> WikiText-2 PPL: 11.05<br> C4 PPL: 12.99
Loading
model_path = "radi-cho/gemma-2-2b-AWQ"
# With transformers
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(model_path, device_map="cuda:0")
# With transformers (fused)
from transformers import AutoModelForCausalLM, AwqConfig
quantization_config = AwqConfig(bits=4, fuse_max_seq_len=512, do_fuse=True)
model = AutoModelForCausalLM.from_pretrained(model_path, quantization_config=quantization_config).to(0)
# With AutoAWQ
from awq import AutoAWQForCausalLM
model = AutoAWQForCausalLM.from_quantized(model_path)
- Task
- Text Generation
- Library
- transformers
- Type
- gemma2
- License
- gemma
- Created
- Aug 1, 2024
- Updated
- Aug 1, 2024