gemma-4-e4b-it-VQ-PLE

gemma-4-e4b-it-VQ-PLE

7.39 GiB — the community 8-bit, with a better embedding table. This is mlx-community's gemma-4-e4b-it-8bit with exactly one change: the 5.25 GiB per-layer-embedding table (35% of the model's bytes) is replaced by a vector-quantized version at 5.75 bits/weight. Everything else — attention, MLPs, norms, towers — is byte-identical to the artifact you already know.

That one swap makes the model measurably closer to bf16 than the 8-bit it came from, at 1 GiB less disk, ≈1 GB less peak RAM, and decode speed at parity.

Changelog

2026-10-01 — runtime refresh; Knurlogic

The bundled model.py is updated to the current VQ runtime; the weights are unchanged. The run, multi-Mac and vision instructions now use Knurlogic.

2026-09-09 — runtime refresh

Runtime refresh. The bundled model.py is updated so downloaders run exactly the code that was benchmarked; dense bundles now carry both runtimes.

• Faster prefill on affected geometries, measured per rung: a device-codebook kernel arm for large-codebook geometries (up to 1.46x on affected rungs), a ragged-subvector relaxation (up to 1.34x on affected rungs), a fused d8 arm, and a routing memo (≈2%). No blanket speedup is claimed across the lineup — gains apply only where the geometry engages the new paths.
• Output quality is unchanged: the kernel changes are bit-identical or 1-ULP-equivalent, and the routing memo is bit-identical (logits checksum verified).
• Speculative decoding (MTP): on repos that ship mtp-head-q6.safetensors, the sidecar works with the exo fork branch mtp-stage1 (github.com/noahzelezny/exo) — launch each node with exo --mtp (or set EXO_MTP=1). Note: with MTP enabled, exo serves requests sequentially (the batch engine has no MTP path), so leave it off for concurrent / multi-agent workloads.

2026-09 — bundle repair

Bundle repair — this model was broken for downloaders until today.

Two stacked defects, both ours, both fixed:

1. The shipped model.py was incomplete (it imported runtime modules that only existed in our development environment), so on a stock install the model could not load at all. Repairing that exposed:
2. The VQ embedding decode returned fp16 into a bf16 model; MLX's promotion rules turned the whole forward fp32 from layer 1 on, which at this model's attention geometry crashes the Metal kernel on any real prompt. One cast fixes it.

No published score was ever affected — scoring ran through the correct decode path; only the bundled runtime was broken. The bundle is now self-contained, gated by a load-and-generate check in a verified-clean environment, and generates coherently at length on both M3- and M4-class machines.

Peak memory, measured externally: process RSS sampled at 5 Hz across load + 2048-token prefill + 128-token decode peaks at 7.5 GiB — the on-disk size. Peak ≈ resident.

Measured results

All numbers on the same instruments, same corpus, same teacher cache; "incumbent" = mlx-community gemma-4-e4b-it-8bit as shipped.

Runtime rows measured 2026-09 on an M3 Ultra (mlx 0.32.2, 256-token greedy decode, repeated runs within 0.6%). The 2026-09 bundle's kernel refresh closed what used to be an ≈8% decode deficit against the 8-bit.

Honest summary: closer to the bf16 teacher on the precise instrument (KL), indistinguishable on the noisy one (litbench, n=104 cannot resolve a 3-point gap — SE ±3.7), decode at parity, ≈12% less RAM, 1 GiB less disk. There is no axis left on which the 8-bit is the better download.

Speed

Measured 2026-10-01 on this revision.

One Mac Studio M3 Ultra, vqlab speed-pair: a fresh process per run, alternating with the reference, 3 pairs, 2048-token prompt, 128 generated tokens, no drafting.

Throughput differs across Apple silicon; compare the ratio, not the absolute numbers.

Run it

Requires a Mac with Apple Silicon (macOS, MLX).

With Knurlogic, which works out the settings and checks the model fits before loading it:

pip install knurlogic
hf download TheDrainFlorist/gemma-4-e4b-it-VQ-PLE --local-dir ~/Knurlogic/Models/gemma-4-e4b-it-VQ-PLE
knurlogic serve ~/Knurlogic/Models/gemma-4-e4b-it-VQ-PLE

Chat at http://127.0.0.1:8080/, or point any OpenAI-, Anthropic- or Ollama-compatible client at it (model local). knurlogic ui opens a page with a Launch button for every model on disk instead.

Images work through Knurlogic directly; mlx-vlm is not needed.

The VQ runtime ships inside this repo as model.py (declared by model_file in config.json), and Knurlogic runs the file the model ships.

Without Knurlogic:

pip install 'mlx-lm>=0.31.3'
mlx_lm.chat --model TheDrainFlorist/gemma-4-e4b-it-VQ-PLE

Why the embedding table

We first vector-quantized everything (MLPs + embeddings) and LOST to the 8-bit decisively (20.8 vs 8.1 mnats): e4b's MLP weights do not tolerate 5.75-bit VQ even though their reconstruction error looks excellent — fit error and output damage are different quantities. An ablation split the damage: the MLP contributed essentially all of it, and the VQ embedding table alone was better than its 8-bit affine counterpart. So this artifact keeps the 8-bit MLPs and ships only the swap that wins.

Embeddings are the friendly case for VQ at runtime too: a lookup decodes only the rows a batch touches — no matmul kernel, no full-table materialization, which is where the RAM saving comes from.

No calibration data

The VQ fit is pure weight-space k-means against the bf16 tensors: no calibration corpus, no forward passes, no distillation. 154 seconds of codebook fitting on an M3 Ultra. Every number above was measured after, not optimized for.

Methodology

Fitted data-free from the bf16 checkpoint — k-means / Lloyd over weight subvectors, seed 1234, no Hessian, no activation statistics, no calibration corpus. Recipes are in the VQLab repo.

Only the per-layer-embedding table is replaced; every other tensor is carried over from the 8-bit conversion unchanged. See Why the embedding table above for why that is the high-leverage target.

Verification

• VQ table verified decode-side against the bf16 source (relerr 0.0296, uniform across all 262,144 rows).
• Packed codes verified bit-exact against the unpacked reference, and the packed artifact reproduces the KL score to the third decimal.
• Corrupting the codebook garbles generation — the VQ path is provably live, not a fallback.

Limitations

• ≈8% slower decode and ≈20% slower prefill than the 8-bit incumbent.
• litbench point estimate is 3 points below the incumbent; the paired test says noise (p=0.45), and the KL says closer-to-teacher, but if your use case resembles literary MC comprehension specifically, measure your own.
• Vision/audio towers are carried unchanged from the incumbent artifact; vision was not re-benched here.

Paper

The method, the full model ladder, the negative results, and the measurement rules behind every number here: Data-Free Vector Quantization Beats Affine Quantization at Matched Bytes Below 6 Bits (CC BY 4.0) · code: VQLab · web version: Space

Support this work

VQLab and these artifacts are built and released independently — the fits, the measurement harness and the published rungs are one person's compute and time. If they are useful to you:

• Sponsor: github.com/sponsors/noahzelezny
• Contract work: available for quantization and on-device inference work on Apple silicon — custom rungs, per-layer allocation for your model, or getting a checkpoint to run well on a Mac. Contact: [email protected]

Provenance

Base model: google/gemma-4-e4b-it — governed by the Gemma Terms of Use. This is a quantized derivative and inherits that license; using it means accepting the base model's terms. Quantization: TheDrainFlorist, 2026.

This build starts from mlx-community/gemma-4-e4b-it-8bit and replaces its per-layer-embedding table; that conversion's authors are credited accordingly.

Built with MLX and VQLab.