mlx-quant-ladder-findings

MLX Quantisation Ladders — Measurements

Five uncensored model families, each converted from one BF16 source with one group size, so bit width is the only variable within a family. Converted with mlx_vlm.convert, measured on a single Apple M3 Ultra (96 GB unified, macOS 27).

Perplexity: allenai/tulu-3-sft-mixture, 192 samples x 512 tokens, seed 123 — identical for every rung. Throughput: aggregate output tok/s on ~70-token prompts, with a unique nonce per request so no measurement reads another's prefix cache (cached_tokens asserted 0 on every level).

Perplexity is comparable only WITHIN a family. Tokenizers differ across families (Gemma 262,144 vocab vs Qwen 151,936), so a cross-family comparison of these numbers is meaningless. The x best column is the portable one.

Muse-Glimmer-30B-Abliterated

Qwen3.8-27B-Abliterated

Ornith-1.5-9B-Abliterated

Gemma-4-12B-Abliterated

Gemma-4-26B-A4B-Heretic

What the numbers say

4-bit is free on the Qwen lineage. Qwen3.8-27B, Ornith-1.5-9B and Muse-Glimmer-30B all sit within ~1.03x of their best rung from 4-bit upward. 8-bit costs 70%+ more disk for no measurable quality gain.

Gemma is quantisation-fragile. Both Gemma-4 families degrade noticeably at 4-bit (1.67x and 2.06x) and collapse at 3-bit (5.5x and 976x) under the identical pipeline that leaves Qwen models flat. That is a property of the architecture, not of the tooling.

MXFP4 is often the best rung available. Best-in-family on Qwen3.8-27B (1.00x), and on Muse-Glimmer it was simultaneously the fastest rung measured and the second-smallest.

2-bit was built, measured and deleted. Across four families it ran 15x to 5e9x its family best while saving barely a gigabyte over 3-bit. No 2-bit artifacts are published.

Selection rule

A rung is published only if:

1. its weight index exists and every shard it names is present and non-trivial, and
2. its perplexity is <= 2x the best rung in its own family.

Five rungs were withheld. Each withheld rung's ratio appears in the tables above, so the decision is auditable rather than asserted.

Loading

These are mlx-vlm models. mlx_lm cannot load muse_glimmer or gemma4_unified at all — those architectures are registered only in mlx-vlm.

pip install mlx-vlm
mlx_vlm.generate --model shoemoney/<repo> --prompt "Hello" --max-tokens 256