gemma-4-31B-it-oQ6

gemma-4-31B-it-oQ6

An oQ6 mixed-precision quantization of google/gemma-4-31b-it using oMLX — a data-driven, sensitivity-aware quantization system for Apple Silicon.

Produces standard MLX safetensors compatible with oMLX, LM Studio, mlx-lm, and any MLX-compatible inference server.

Key Facts

Why oQ6?

oQ6 fills the gap between the existing oQ4 (~18 GB) and oQ8 (~31 GB) variants. At ~25 GB, it fits comfortably on 32 GB MacBooks with room for KV cache, while maintaining high quality with vision capabilities intact.

oQ measures per-layer quantization sensitivity through calibration inference and allocates bits where they matter most. At 6-bit, quality remains very close to BF16 — significantly better than 4-bit, with meaningful speed improvements over 8-bit.

Benchmarks

Tested on Apple M2 Ultra (128 GB, 76 GPU cores) with oMLX. Generation length: 128 tokens.

oQ6 (this model, 25.2 GB)

Continuous Batching (pp1024/tg128)

Comparison: oQ6 vs oQ8 vs BF16

oQ6 is the sweet spot for 32 GB Macs: fits with room for KV cache, 9% faster generation than oQ8, and quality remains close to near-lossless.

Usage

oMLX

Drop the model folder into your oMLX models directory. Auto-detected on server start.

mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("mpe74/gemma-4-31B-it-oQ6")
messages = [{"role": "user", "content": "Your prompt here"}]
prompt = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
response = generate(model, tokenizer, prompt=prompt, max_tokens=2048)

Recommended Sampling Parameters

LM Studio

Search for the model and download. Works with MLX backend on Apple Silicon.

Quantization Details

Also Available

• mpe74/gemma-4-31B-it-oQ8 — Near-lossless 8-bit (~31 GB) for 64+ GB Macs

---

Quantized by mpe74 using oMLX on Apple M2 Ultra (128 GB).