Downloads · 30 days
26
3% of all-time downloads
gemma-4-31B-it-oQ8
mpe74/gemma-4-31B-it-oQ8
gemma-4-31B-it-oQ8 is a image-text-to-text model from mpe74. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for mlx. The card lists the license as gemma.
An oQ8 mixed-precision quantization of google/gemma-4-31b-it using oMLX — a data-driven, sensitivity-aware quantization system for Apple Silicon.
All-time downloads
962
Public
Parameters
31.3B
33.8 GB on disk
Likes
1
Public
Repo makeup
Click a slice to open those files.
.safetensors33.8 GB · 100%
Parameter types
How the weights are stored.
U3230.7B · 98%
Try a prompt
Base models
Model card
From the Hugging Face model README
gemma-4-31B-it-oQ8
An oQ8 mixed-precision quantization of google/gemma-4-31b-it using oMLX — a data-driven, sensitivity-aware quantization system for Apple Silicon.
Produces standard MLX safetensors compatible with oMLX, LM Studio, mlx-lm, and any MLX-compatible inference server.
Key Facts
| Property | Value |
|---|---|
| Base Model | google/gemma-4-31b-it (31B dense, BF16) |
| Quantization | oQ8 — sensitivity-driven mixed-precision |
| Effective bpw | 8.6 |
| Model Size | ~31.4 GB (vs. 58.3 GB BF16) |
| Vision | ✅ Preserved (vision weights kept in fp16) |
| Format | Standard MLX safetensors |
| Quantized with | oMLX v0.3.4+ |
| Hardware | Apple M2 Ultra 128 GB |
Why oQ8?
oQ is not uniform quantization. Instead of applying the same bit depth to every layer, oQ measures per-layer quantization sensitivity through calibration inference and allocates bits where they matter most. Critical layers (embeddings, LM head, sensitive transformer layers) receive higher precision while the bulk of weights are quantized to the target bit depth.
At 8-bit, this is near-lossless quality at roughly half the size of BF16 — with significantly faster token generation due to reduced memory bandwidth requirements on Apple Silicon.
Benchmarks
Tested on Apple M2 Ultra (128 GB) with oMLX. Generation length: 128 tokens.
oQ8 (this model, 31.8 GB)
| Test | TTFT (ms) | TPOT (ms) | pp TPS | tg TPS | E2E (s) | Throughput | Peak Mem |
|---|---|---|---|---|---|---|---|
| pp1024/tg128 | 5,777 | 57.4 | 177.3 tok/s | 17.5 tok/s | 13.1s | 88.1 tok/s | 31.80 GB |
| pp4096/tg128 | 22,680 | 63.6 | 180.6 tok/s | 15.8 tok/s | 30.8s | 137.3 tok/s | 33.65 GB |
| pp8192/tg128 | 45,656 | 73.2 | 179.4 tok/s | 13.8 tok/s | 55.0s | 151.4 tok/s | 33.96 GB |
Continuous Batching (pp1024/tg128)
| Batch | tg TPS | Speedup | pp TPS | pp TPS/req | Avg TTFT (ms) | E2E (s) |
|---|---|---|---|---|---|---|
| 1x (baseline) | 17.5 tok/s | 1.00x | 177.3 tok/s | 177.3 tok/s | 5,777 | 13.1 |
| 2x | 28.9 tok/s | 1.65x | 176.6 tok/s | 88.3 tok/s | 11,402 | 20.5 |
| 4x | 37.0 tok/s | 2.11x | 176.1 tok/s | 44.0 tok/s | 22,529 | 37.1 |
BF16 reference (58.5 GB)
| Test | TTFT (ms) | TPOT (ms) | pp TPS | tg TPS | E2E (s) | Throughput | Peak Mem |
|---|---|---|---|---|---|---|---|
| pp1024/tg128 | 3,974 | 98.2 | 257.7 tok/s | 10.3 tok/s | 16.4s | 70.1 tok/s | 58.53 GB |
| pp4096/tg128 | 16,061 | 102.9 | 255.0 tok/s | 9.8 tok/s | 29.1s | 145.0 tok/s | 60.31 GB |
| pp8192/tg128 | 32,348 | 115.2 | 253.2 tok/s | 8.8 tok/s | 47.0s | 177.1 tok/s | 60.63 GB |
Continuous Batching (pp1024/tg128)
| Batch | tg TPS | Speedup | pp TPS | pp TPS/req | Avg TTFT (ms) | E2E (s) |
|---|---|---|---|---|---|---|
| 1x (baseline) | 10.3 tok/s | 1.00x | 257.7 tok/s | 257.7 tok/s | 3,974 | 16.4 |
| 2x | 9.1 tok/s | 0.88x | 256.8 tok/s | 128.4 tok/s | 7,713 | 36.1 |
| 4x | 16.7 tok/s | 1.62x | 253.1 tok/s | 63.3 tok/s | 15,215 | 46.8 |
Summary: oQ8 vs BF16
| Metric | oQ8 | BF16 | Difference |
|---|---|---|---|
| Size | 31.4 GB | 58.3 GB | -46% |
| Token Generation | 17.5 tok/s | 10.3 tok/s | +70% faster |
| 4x Batch Generation | 37.0 tok/s | 16.7 tok/s | +122% faster |
| Prefill | 177 tok/s | 258 tok/s | -31% (dequantization overhead) |
| Peak Memory | 31.8 GB | 58.5 GB | -46% |
oQ8 is near-lossless at half the memory, with significantly faster token generation. The prefill speed is slightly slower due to dequantization, but for interactive use the generation speed is what matters.
Usage
oMLX
Drop the model folder into your oMLX models directory. Auto-detected on server start.
mlx-lm
from mlx_lm import load, generate
model, tokenizer = load("mpe74/gemma-4-31B-it-oQ8")
messages = [{"role": "user", "content": "Your prompt here"}]
prompt = tokenizer.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True
)
response = generate(model, tokenizer, prompt=prompt, max_tokens=2048)
LM Studio
Search for the model and download. Works with MLX backend on Apple Silicon.
Quantization Details
| Parameter | Value |
|---|---|
| oQ Level | oQ8 |
| Base bits | 8 |
| Mode | Affine quantization |
| Group size | 64 |
| Sensitivity model | Source model (google/gemma-4-31b-it BF16) |
| Calibration data | Built-in oMLX dataset (600 samples: code, multilingual, tool calling, reasoning) |
| Vision weights | Preserved in fp16 |
Bug Note
During quantization, a bug in oMLX was encountered: Object of type set is not JSON serializable caused by _oq_non_quantizable (a Python set) not being removed from the output config before JSON serialization. Fix: add "_oq_non_quantizable" to the cleanup list in omlx/oq.py line ~1300. Issue will be reported upstream.
Quantized by mpe74 using oMLX on Apple M2 Ultra (128 GB).
- Task
- Image-Text-to-Text
- Library
- mlx
- Type
- gemma4
- License
- gemma
- Created
- Apr 7, 2026
- Updated
- Apr 7, 2026