gemma-4-26b-a4b-it-VQ-6.2bpw

gemma-4-26b-a4b-it-VQ-6.2bpw

18.7 GiB — indistinguishable from bf16 on blind literary judging, at 39% of bf16's size. Vision included.

A vector-quantized build of gemma-4-26b-a4b-it for Apple Silicon. Stock mlx-lm, no patches — the VQ runtime ships inside the checkpoint as model.py.

Changelog

2026-10-01 — runtime refresh; Knurlogic

The bundled model.py is updated to the current VQ runtime; the weights are unchanged. The run, multi-Mac and vision instructions now use Knurlogic.

2026-09-09 — runtime refresh

Runtime refresh. The bundled model.py is updated so downloaders run exactly the code that was benchmarked; dense bundles now carry both runtimes.

• Faster prefill on affected geometries, measured per rung: a device-codebook kernel arm for large-codebook geometries (up to 1.46x on affected rungs), a ragged-subvector relaxation (up to 1.34x on affected rungs), a fused d8 arm, and a routing memo (≈2%). No blanket speedup is claimed across the lineup — gains apply only where the geometry engages the new paths.
• Output quality is unchanged: the kernel changes are bit-identical or 1-ULP-equivalent, and the routing memo is bit-identical (logits checksum verified).
• Speculative decoding (MTP): on repos that ship mtp-head-q6.safetensors, the sidecar works with the exo fork branch mtp-stage1 (github.com/noahzelezny/exo) — launch each node with exo --mtp (or set EXO_MTP=1). Note: with MTP enabled, exo serves requests sequentially (the batch engine has no MTP path), so leave it off for concurrent / multi-agent workloads.

2026-09 — bundle refresh

Bundle refresh.

model.py updated as part of the 2026-09 refresh shared across the VQ lineup: updated expert kernels (outputs verified equivalent), a default buffer-cache ceiling so transient prefill allocations return to the OS as they free, and dual-runtime loading (mlx-lm and mlx_vlm). Weights unchanged; prior revision pinnable.

Peak memory, measured externally: process RSS sampled at 5 Hz from outside across load + 2048-token prefill + 128-token decode peaks at 18.8 GiB — the on-disk size. Peak ≈ resident.

Measured results

Perplexity is NOT reported here, and that is deliberate. Raw loglikelihood is invalid on the gemma-4-it family — the RL-sharpened distribution collapses, so wikitext ppl reads ≈100-700 while the model generates fluent prose. Verified independently against HF transformers on unquantized bf16, so it is a model property, not an MLX artifact. Anyone quoting gemma-4 perplexity is quoting noise.

Instead: blind pairwise judging against bf16, plus KL-to-bf16.

Blind literary judging vs bf16 (the headline)

60 literary continuations, greedy. Pairs anonymized with A/B randomized per pair and the key withheld from the judge (claude-sonnet-5), which was told ties are legitimate. Exact two-sided sign test on decisive pairs.

Statistically indistinguishable from bf16, with the judge unable to separate the texts in 43% of pairs. Read that as "indistinguishable", NOT "better than bf16" — the judge showed a positional lean and bf16 sat in the disfavored slot, so the direction is not claimable.

For contrast, the same instrument on weaker builds of this model: a 12.5 GiB d4 build lost 36-20 (p=0.044) and a 9.4 GiB build lost 34-12 (p=0.0016). Both are significantly worse than bf16. This one is not.

KL divergence to bf16

80% is the practical ceiling here, not 100%. This is a 128-expert MoE with top-8 routing: any perturbation flips which experts fire, so even a near-lossless 8-bit quant only reproduces bf16's argmax ≈80% of the time. Judge against the 8-bit row, never against 100%.

Choosing a size (read this before downloading)

This artifact exists for one job: bf16-class quality in the smallest size that actually delivers it (18.7 GiB; blind judges could not tell it from bf16). Below this size, quantizing the 26B harder stops making sense — we measured our own smaller 26B builds against mlx-community/gemma-4-e4b-it-8bit across paired litbench, KL, constraint benches, and a capability ladder, and the honest result is that at small sizes the e4b is the right model: it matches or beats aggressive 26B quants at similar or smaller size, and it is faster.

So the family guidance is:

• ≈19 GiB budget, want bf16-class quality or long-form generation (2,500+ words sustained): this artifact.
• ≈8 GiB budget: use gemma-4-e4b-it-8bit — not a compromise; it wins that bracket on our measurements.
• Tighter than 8 GiB or RAM-bound: see our VQ-PLE build of the e4b (7.39 GiB, measurably closer to bf16 than the 8-bit, 20% less RAM, ≈8% slower decode) — the right pick when memory is the constraint and latency is not.

Runtime

Measured on an M4 Max (128 GB), macOS, stock mlx-lm, GPU otherwise idle, 120-token greedy generation:

Fits a 32 GB machine with room for context.

Run it

Requires a Mac with Apple Silicon (macOS, MLX).

With Knurlogic, which works out the settings and checks the model fits before loading it:

pip install knurlogic
hf download TheDrainFlorist/gemma-4-26b-a4b-it-VQ-6.2bpw --local-dir ~/Knurlogic/Models/gemma-4-26b-a4b-it-VQ-6.2bpw
knurlogic serve ~/Knurlogic/Models/gemma-4-26b-a4b-it-VQ-6.2bpw

Chat at http://127.0.0.1:8080/, or point any OpenAI-, Anthropic- or Ollama-compatible client at it (model local). knurlogic ui opens a page with a Launch button for every model on disk instead.

Images work through Knurlogic directly; mlx-vlm is not needed.

The VQ runtime ships inside this repo as model.py (declared by model_file in config.json), and Knurlogic runs the file the model ships.

Without Knurlogic:

pip install 'mlx-lm>=0.31.3'
python -m mlx_lm generate \
  --model TheDrainFlorist/gemma-4-26b-a4b-it-VQ-d2K2048 \
  --prompt "Continue this passage in Austen's voice: ..." \
  --max-tokens 400

Methodology

Experts are vector-quantized at d=2, K=2048 (5.75 bits/weight packed); everything else — attention, router, embeddings, the per-layer dense MLP — stays at 8-bit. Experts are ≈90% of the language model, so that is where the bytes are.

d=2 matters: at d=4 every +0.25 bpw costs a codebook DOUBLING, so d4 saturates near 3.5 bpw and 61% agreement. Reaching this quality at d=4 would need K=2^22. Halving d is the only way to spend enough bits to approach the ceiling.

Verification

Every tensor was decoded from the published artifact (not from a fit log) and compared against the bf16 source; no tensor exceeds 3x the artifact's own median reconstruction error. This caught real silent corruption during development — a fit log reporting healthy error while the written file held destroyed tensors — so it is a standing gate, not a formality.

Limitations

• Do not benchmark this with perplexity. See above.
• Top-1 agreement with bf16 is 77.89%, versus 79.95% for a 25 GiB 8-bit build. The blind judging says readers cannot tell, but the token-level divergence is real and larger than 8-bit's.
• Blind judging is one judge, one pass, n=60 with 26 ties — as few as 34 decisive pairs. The ordering across builds is robust; the absolute margin is not tightly bounded.
• Audio is not included: this checkpoint ships zero audio tensors upstream.

Paper

The method, the full model ladder, the negative results, and the measurement rules behind every number here: Data-Free Vector Quantization Beats Affine Quantization at Matched Bytes Below 6 Bits (CC BY 4.0) · code: VQLab · web version: Space

Support this work

VQLab and these artifacts are built and released independently — the fits, the measurement harness and the published rungs are one person's compute and time. If they are useful to you:

• Sponsor: github.com/sponsors/noahzelezny
• Contract work: available for quantization and on-device inference work on Apple silicon — custom rungs, per-layer allocation for your model, or getting a checkpoint to run well on a Mac. Contact: [email protected]

Provenance

Base model: google/gemma-4-26b-a4b-it — governed by the Gemma Terms of Use. This is a quantized derivative and inherits that license; using it means accepting the base model's terms. Quantization: TheDrainFlorist, 2026.

Built with MLX and VQLab.