Downloads · 30 days
9
8% of all-time downloads
granite-4.1-8b-8bit-mlx
sahilchachra/granite-4.1-8b-8bit-mlx
granite-4.1-8b-8bit-mlx is a machine learning model from sahilchachra. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for mlx. The card lists the license as apache-2.0.
Quantized version of ibm-granite/granite-4.1-8b for Apple Silicon using MLX.
All-time downloads
120
Public
Parameters
8.4B
8.9 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors8.9 GB · 100%
Parameter types
How the weights are stored.
U328.4B · 100%
Base models
Model card
From the Hugging Face model README
granite-4.1-8b-8bit-mlx
Quantized version of ibm-granite/granite-4.1-8b for Apple Silicon using MLX.
Quantization: Affine integer quantization
Precision: 8-bit (~8.5 bits/weight avg)
Group size: 64
Disk size: 8499 MB
Quantized by: sahilchachra
About this variant
Affine quantization at 8-bit with group size 64. Closest to FP16 quality. Recommended when memory allows and quality is the priority.
Benchmark results
Evaluated on Apple M5 Pro with MLX. All metrics measured in a single pass (model loaded once).
Performance
| This model | FP16 baseline | |
|---|---|---|
| Prefill (tok/s) | 438.5 | 334.25 |
| Decode (tok/s) | 32.93 | 17.88 |
| Peak memory (GB) | 9.783 | 17.549 |
| Disk size (MB) | 8499 | 16778 |
Quality
| Benchmark | This model | FP16 baseline | Task |
|---|---|---|---|
| GSM8K | 93.3% | 90.0% | Math reasoning (25 samples) |
| MMLU | 56.0% | 60.0% | World knowledge (50 samples) |
| HumanEval | 33.3% | 30.0% | Code pass@1 (20 samples) |
Context scaling (decode tok/s)
| Context length | Decode tok/s |
|---|---|
| ~128 tokens | 31.8 |
| ~256 tokens | 31.7 |
| ~512 tokens | 31.7 |
| ~1024 tokens | 31.5 |
Usage
Install
pip install mlx-lm
Generate
from mlx_lm import load, generate
model, tokenizer = load("sahilchachra/granite-4.1-8b-8bit-mlx")
response = generate(model, tokenizer, prompt="Your prompt here", max_tokens=512, verbose=True)
Stream
from mlx_lm import load, stream_generate
model, tokenizer = load("sahilchachra/granite-4.1-8b-8bit-mlx")
for chunk in stream_generate(model, tokenizer, prompt="Your prompt here", max_tokens=512):
print(chunk.text, end="", flush=True)
All variants in this collection
| Model | Method | Bits/weight |
|---|---|---|
| sahilchachra/granite-4.1-8b-4bit-mlx | Affine int4 (group 64) | |
| sahilchachra/granite-4.1-8b-5bit-mlx | Affine int5 (group 64) | |
| sahilchachra/granite-4.1-8b-6bit-mlx | Affine int6 (group 64) | |
| sahilchachra/granite-4.1-8b-8bit-mlx | Affine int8 (group 64) | ← this model |
| sahilchachra/granite-4.1-8b-mixed4_6-mlx | Mixed 4+6 bit | |
| sahilchachra/granite-4.1-8b-mxfp4-mlx | Block float MX FP4 | |
| sahilchachra/granite-4.1-8b-mxfp8-mlx | Block float MX FP8 |
Notes
- Requires Apple Silicon (M1 or later) with MLX
- Benchmarks run on Apple M5 Pro, 24 GB unified memory
- Sample sizes are small (25–50 per benchmark) — treat accuracy figures as indicative, not definitive
- Base model license: Apache 2.0
Original model
See ibm-granite/granite-4.1-8b for full model details, training information, and intended use.
- Library
- mlx
- Type
- granite
- License
- apache-2.0
- Languages
- en
- Created
- May 22, 2026
- Updated
- May 22, 2026