Downloads · 30 days
21
3% of all-time downloads
gemma-4-4bit-it-mlx-dwq-ram-5gb
chantra/gemma-4-4bit-it-mlx-dwq-ram-5gb
gemma-4-4bit-it-mlx-dwq-ram-5gb is a text generation model from chantra. Use it when you need the model to write or continue text. It is set up for mlx.
4-bit DWQ (Distillation + Weight Quantization) of gemma-4-E4B-it for Apple Silicon via MLX. Quantized from an 8-bit teacher. Fits comfortably in 5 GB RAM with better perplexity than standard 4-bit.
All-time downloads
608
Public
Parameters
7.5B
4.3 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors4.2 GB · 99%
Parameter types
How the weights are stored.
U327.5B · 100%
Try a prompt
Base models
Model card
From the Hugging Face model README
Gemma 4 E4B IT — DWQ 4-bit (MLX, ~5 GB RAM)
4-bit DWQ (Distillation + Weight Quantization) of gemma-4-E4B-it for Apple Silicon via MLX.
Quantized from an 8-bit teacher. Fits comfortably in 5 GB RAM with better perplexity than standard 4-bit.
Model Details
| Property | Value |
|---|---|
| Base model | google/gemma-4-e4b-it |
| Parameters | 7.5B |
| Quantization | DWQ 4-bit (teacher: 8-bit) |
| Framework | MLX |
| RAM | ~5 GB |
Evaluation Results
ARC Easy
| Metric | Score |
|---|---|
| Accuracy | 0.7113 |
| Accuracy (normalized) | 0.6540 |
Perplexity
- Dataset:
allenai/tulu-3-sft-mixture(256 samples, 131,072 tokens, seq len 512) - Perplexity: 42.847 ± 0.638
- Speed: 2002 tokens/sec
- Time: 65.35s
- Peak Memory: 11.25 GB
TruthfulQA
Generation:
| Metric | Score |
|---|---|
| BLEU Max | 9.69 |
| BLEU Acc | 0.52 |
| ROUGE-1 Acc | 0.56 |
| ROUGE-2 Acc | 0.44 |
| ROUGE-L Acc | 0.55 |
Multiple Choice:
| Metric | Score |
|---|---|
| MC1 Accuracy | 0.31 |
| MC2 Accuracy | 0.5637 |
Comparison vs Other Quantizations
| Model | ARC Easy (acc) | Perplexity ↓ | TruthfulQA MC2 | RAM |
|---|---|---|---|---|
| 8-bit (base) | 0.7168 | 44.44 | 0.5705 | 8.5 GB |
| DWQ 4-bit (this) | 0.7113 | 42.85 ✅ | 0.5637 | 4.8 GB |
| Std 4-bit | 0.7075 | 54.07 ❌ | 0.5506 | 4.8 GB |
DWQ 4-bit achieves lower perplexity than 8-bit while cutting memory by ~4 GB. Standard 4-bit shows significant perplexity degradation (54.07) in comparison.
Usage
from mlx_lm import load, generate
model, tokenizer = load("chantra/gemma-4-4bit-it-mlx-dwq-ram-5gb")
response = generate(model, tokenizer, prompt="Hello!", max_tokens=512)
print(response)
- Task
- Text Generation
- Library
- mlx
- Type
- gemma4
- Languages
- en
- Created
- Apr 24, 2026
- Updated
- Apr 27, 2026