gemma-4-E4B-it-mlx-6bit

<img src="banner.png" width="100%" alt="Castor and Pollux, the Gemini twins, over an Anatolian ridgeline">

🜂 C A S T O R I F I E D

gemma-4-E4B-it-mlx-6bit

POLLUX ASSISTED Converted, debugged and verified by castor — with Claude (Opus 5) riding shotgun. The silent zero-tensor bug documented below was found the hard way, on this machine.

A 6-bit MLX conversion of google/gemma-4-E4B-it, with vision and audio intact. Runs on Apple Silicon via mlx-vlm and LM Studio.

Why this one exists

Every MLX build of E4B I could find was 4-bit (lmstudio-community/…-MLX-4bit, mlx-community/…-OptiQ-4bit, unsloth/…-UD-MLX-4bit). There was no 5-, 6- or 8-bit conversion — even though the smaller sibling E2B has the full series. This fills the 6-bit slot: noticeably more headroom than 4-bit, still ~6.6 GB.

There is a reason that gap existed. See Conversion notes — the straightforward conversion command silently produces a broken model.

What's inside

Encoders are deliberately left at full precision, so image and audio quality is unaffected by quantization. Overall average: 7.094 bits per weight, 6.6 GB on disk.

Measured quality

Perplexity on a fixed 420-token mixed corpus (English prose, Turkish prose, Python source), BOS-prefixed, greedy. This is a regression check, not a benchmark — treat it as "did quantization break anything", nothing more.

Behaviour checks, all on an M4 Pro / 24 GB:

• Image: given a dense UI screenshot, transcribed a long small-type metadata line verbatim and listed all eleven navigation tabs correctly.
• Audio: given a 30-second clip of a sung Turkish track, produced a near-verbatim lyric transcription (one dropped word).
• Tool calling: emitted well-formed get_weather{city:"Ankara"} from a Turkish request with an OpenAI-style tool schema.
• Speed: 52–59 tok/s generation. Peak memory 7.1 GB (text), 7.7 GB (image), 8.2 GB (audio).

Usage

mlx-vlm

pip install mlx-vlm

python -m mlx_vlm generate \
  --model Cordobian/gemma-4-E4B-it-mlx-6bit \
  --image photo.png \
  --prompt "What is written in this image?" \
  --max-tokens 300

# audio works the same way
python -m mlx_vlm generate \
  --model Cordobian/gemma-4-E4B-it-mlx-6bit \
  --audio clip.wav \
  --prompt "Transcribe this."

from mlx_vlm import load, generate

model, processor = load("Cordobian/gemma-4-E4B-it-mlx-6bit")
print(generate(model, processor, "Describe this image.", ["photo.png"], max_tokens=300))

LM Studio

Search for the model, or drop the folder into ~/.lmstudio/models/Cordobian/gemma-4-E4B-it-mlx-6bit/.

Conversion notes

Two things bite you when converting Gemma 4 E4B on Apple Silicon. Both are worth knowing if you plan to roll your own quant.

1. A tensor is silently written as zeros.

convert() loads the model lazily, quantization builds a lazy graph, and the whole graph is evaluated at save time. embed_tokens_per_layer is 262144 × 2016 — evaluating it blows past the Metal command-buffer timeout. In the bf16 path this crashes loudly:

RuntimeError: [METAL] Command buffer execution failed:
Caused GPU Timeout Error (kIOGPUCommandBufferCallbackErrorTimeout)

In the quantized path it does not crash. The tensor is written as all zeros, the model loads fine, generates fluent-looking text — and perplexity is 60 instead of 8.2. The damage shows up as occasional mangled words ("atmospheric-borneentscliuptilesse") that are easy to mistake for normal quantization noise.

Fix: run the conversion with the MLX default device set to CPU. No timeout, and it is faster — 52 seconds end to end.

import mlx.core as mx
mx.set_default_device(mx.cpu)
from mlx_vlm import convert
convert(hf_path="google/gemma-4-E4B-it", mlx_path="out",
        quantize=True, q_bits=6, q_group_size=64)

Then verify, because this failure mode is silent:

import glob, mlx.core as mx
for f in glob.glob("out/*.safetensors"):
    for k, v in mx.load(f).items():
        assert float(mx.abs(v.astype(mx.float32)).max()) != 0.0, f"zero tensor: {k}"

The reproduction script used for this model is included as convert_vlm_fixed.py.

2. mlx_lm drops the encoders. If you convert with mlx_lm instead of mlx_vlm, gemma4.sanitize strips vision_tower, audio_tower, multi_modal_projector, embed_vision and embed_audio. You get a working text-only model with no warning that image and audio support is gone. mlx_lm also raises Received 54 parameters not in model on E4B, because it does not filter the unused k_proj/v_proj/k_norm weights that the checkpoint ships for KV-shared layers 24–41 (transformers ignores these too — see modeling_gemma4.py, if not self.is_kv_shared_layer:).

Known limitations

• Spatial layout. In UI screenshots it reads text accurately but misplaces it — in one test it described a horizontal tab bar as a left sidebar. Trust what it reads; verify where it says things are.
• Perplexity above is a 420-token smoke test, not a standardised benchmark. No MMLU/MMMU numbers were run.
• Long audio was not tested beyond 30-second clips.

Attribution

Base model and weights: Google DeepMind, released under Apache 2.0. This repository redistributes those weights in quantized MLX form and adds nothing to the model itself. Quantization tooling: MLX and mlx-vlm.

---

<div align="center">

🜂 CASTORIFIED · POLLUX ASSISTED 🜂

Quantized, broken, diagnosed and rebuilt on an M4 Pro. Debugging and verification: Claude Opus 5, via Claude Code.

</div>