Downloads · 30 days
5
14% of all-time downloads
gemma-2-9b-int4
StoyanGanchev/gemma-2-9b-int4
gemma-2-9b-int4 is a machine learning model from StoyanGanchev. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as gemma.
This repository contains a 4-bit quantized version of the Gemma-2-9B model. Designed to minimize memory consumption and speed up inference.
All-time downloads
36
Public
Parameters
9.4B
6.5 GB on disk
Likes
1
Public
Repo makeup
Click a slice to open those files.
.safetensors6.5 GB · 100%
Parameter types
How the weights are stored.
U88.3B · 89%
Model card
From the Hugging Face model README
Gemma-2-9B quantized in Int4
Description
This repository contains a 4-bit quantized version of the Gemma-2-9B model. Designed to minimize memory consumption and speed up inference.
Benchmark results
| Gemma-2-9b | Wiki | C4 | PIQA | ARC-E | ARC-C | HellaSwag | Wino | Avg. |
|---|---|---|---|---|---|---|---|---|
| 0-shot | 0-shot | 0-shot | 0-shot | 25-shot | 0-shot | 0-shot | ||
| Unquantized | 6.88 | 10.12 | 81.39 | 87.25 | 64.33 | 61.27 | 74.11 | 73.67 |
| Int4 | 7.27 | 11.34 | 80.47 | 85.86 | 63.23 | 59.55 | 74.27 | 72.67 |
Benchmark scores are computed with lm-evaluation-harness.
How to use:
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("StoyanGanchev/gemma-2-9b-int4")
model = AutoModelForCausalLM.from_pretrained("StoyanGanchev/gemma-2-9b-int4")
All files could be accesed on repository
- Type
- gemma2
- License
- gemma
- Created
- Aug 2, 2024
- Updated
- Aug 2, 2024