Gemma-4-31B-MLX-4.9bit-vision

spicyneuron/Gemma-4-31B-MLX-4.9bit-vision

Gemma-4-31B-MLX-4.9bit-vision is a image-text-to-text model from spicyneuron. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for mlx. The card lists the license as apache-2.0.

Gemma-4-31B-it optimized for MLX. This quant supports image input and requires a vision-enabled MLX server.

Downloads · 30 days

1% of all-time downloads

All-time downloads

Public

Parameters

31.3B

20 GB on disk

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.safetensors20 GB · 100%

Parameter types

How the weights are stored.

U3230.7B · 98%

Try a prompt

Base models

google/gemma-4-31B-it

Model card

From the Hugging Face model README

Gemma-4-31B-it optimized for MLX. This quant supports image input and requires a vision-enabled MLX server.

Usage

# Start server at http://localhost:8080/chat/completions
uvx --from mlx-vlm --with torchvision \
  mlx_vlm.server \
  --host 127.0.0.1 \
  --port 8080 \
  --model spicyneuron/Gemma-4-31B-MLX-4.9bit-vision

Methodology

Quantized using a custom script inspired by Unsloth/AesSedai/ubergarm style mixed-precision GGUFs. MLX quantization options differ than llama.cpp, but the principles are the same:

Sensitive layers like MoE routing, attention, and output embeddings get higher precision
More tolerant layers like MoE experts get lower precision

Benchmarks

metric	unsloth_gemma-4-31b-it-UD-MLX-4bit	4.9 bit (this model)
bpw	5.765	4.904
prompt processing (1024)	354.269	355.741
token gen (512)	24.842	28.376
peak mem	23.700	20.441
perplexity	35.710 ± 0.201	31.903 ± 0.249
hellaswag	0.53 ± 0.011	0.534 ± 0.011
piqa	0.736 ± 0.01	0.748 ± 0.01
winogrande	0.664 ± 0.013	0.665 ± 0.013

Bits per weight calculated against only the language_model weights.
Perplexity in Gemma 4 was surprisingly high but seemed consistent across my trials. Could be a side effect of using allenai/tulu-3-sft-mixture. Best to interpret it as weaker signal than the other benchmark results.

Tested with:

mlx_lm.benchmark --prompt-tokens 1024 --generation-tokens 512 --num-trials 5
mlx_lm.perplexity --sequence-length 2048 --seed 123
mlx_lm.evaluate --tasks hellaswag --seed 123 --num-shots 0 --limit 2000
mlx_lm.evaluate --tasks piqa --seed 123 --num-shots 0 --limit 2000
mlx_lm.evaluate --tasks winogrande --seed 123 --num-shots 0 --limit 2000

Task: Image-Text-to-Text
Library: mlx
Type: gemma4
License: apache-2.0
Created: Apr 6, 2026
Updated: May 23, 2026