Downloads · 30 days
12
10% of all-time downloads
granite-4.1-8b-mixed4_6-mlx
sahilchachra/granite-4.1-8b-mixed4_6-mlx
granite-4.1-8b-mixed4_6-mlx is a machine learning model from sahilchachra. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for mlx. The card lists the license as apache-2.0.
Quantized version of ibm-granite/granite-4.1-8b for Apple Silicon using MLX.
All-time downloads
123
Public
Parameters
8.4B
5 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors5 GB · 100%
Parameter types
How the weights are stored.
U328.4B · 100%
Base models
Model card
From the Hugging Face model README
granite-4.1-8b-mixed4_6-mlx
Quantized version of ibm-granite/granite-4.1-8b for Apple Silicon using MLX.
Quantization: Mixed-bit quantization (mlx-lm predicate: mixed_4_6)
Precision: ~4.77 bits/weight avg
Group size: 64
Disk size: 4773 MB
Quantized by: sahilchachra
About this variant
Mixed-bit quantization where sensitive layers (embeddings, first/last transformer layers) are kept at 6-bit while the remaining layers use 4-bit. Achieves better quality than uniform 4-bit at nearly the same disk size (~4.8 GB vs ~4.5 GB).
Benchmark results
Evaluated on Apple M5 Pro with MLX. All metrics measured in a single pass (model loaded once).
Performance
| This model | FP16 baseline | |
|---|---|---|
| Prefill (tok/s) | 528.68 | 334.25 |
| Decode (tok/s) | 62.71 | 17.88 |
| Peak memory (GB) | 5.915 | 17.549 |
| Disk size (MB) | 4773 | 16778 |
Quality
| Benchmark | This model | FP16 baseline | Task |
|---|---|---|---|
| GSM8K | 90.0% | 90.0% | Math reasoning (25 samples) |
| MMLU | 54.0% | 60.0% | World knowledge (50 samples) |
| HumanEval | 26.7% | 30.0% | Code pass@1 (20 samples) |
Context scaling (decode tok/s)
| Context length | Decode tok/s |
|---|---|
| ~128 tokens | 53.8 |
| ~256 tokens | 53.5 |
| ~512 tokens | 53.5 |
| ~1024 tokens | 53.0 |
Usage
Install
pip install mlx-lm
Generate
from mlx_lm import load, generate
model, tokenizer = load("sahilchachra/granite-4.1-8b-mixed4_6-mlx")
response = generate(model, tokenizer, prompt="Your prompt here", max_tokens=512, verbose=True)
Stream
from mlx_lm import load, stream_generate
model, tokenizer = load("sahilchachra/granite-4.1-8b-mixed4_6-mlx")
for chunk in stream_generate(model, tokenizer, prompt="Your prompt here", max_tokens=512):
print(chunk.text, end="", flush=True)
All variants in this collection
| Model | Method | Bits/weight |
|---|---|---|
| sahilchachra/granite-4.1-8b-4bit-mlx | Affine int4 (group 64) | |
| sahilchachra/granite-4.1-8b-5bit-mlx | Affine int5 (group 64) | |
| sahilchachra/granite-4.1-8b-6bit-mlx | Affine int6 (group 64) | |
| sahilchachra/granite-4.1-8b-8bit-mlx | Affine int8 (group 64) | |
| sahilchachra/granite-4.1-8b-mixed4_6-mlx | Mixed 4+6 bit | ← this model |
| sahilchachra/granite-4.1-8b-mxfp4-mlx | Block float MX FP4 | |
| sahilchachra/granite-4.1-8b-mxfp8-mlx | Block float MX FP8 |
Notes
- Requires Apple Silicon (M1 or later) with MLX
- Benchmarks run on Apple M5 Pro, 24 GB unified memory
- Sample sizes are small (25–50 per benchmark) — treat accuracy figures as indicative, not definitive
- Base model license: Apache 2.0
Original model
See ibm-granite/granite-4.1-8b for full model details, training information, and intended use.
- Library
- mlx
- Type
- granite
- License
- apache-2.0
- Languages
- en
- Created
- May 22, 2026
- Updated
- May 22, 2026