gemma-4-26B-A4B-it-8bit-XL-mlx

leonsarmiento/gemma-4-26B-A4B-it-8bit-XL-mlx

This model was converted to MLX format from google/gemma-4-26B-A4B-it using BaseQuant_XL 8-bit mixed quantization — the maximum-fidelity XL variant. The vision encoder is preserved (full multimodal). The unquantized BF16 base was used as the source for maximum fidelity.

BaseQuant_XL keeps the most routing-critical layers in full bf16 precision — the MoE router.proj (gate) and the mlp (shared/dense expert that processes all tokens) — while applying 8-bit quantization to everything else. Compared to the 6-bit XL variant, only the routed experts (experts.switch_glu) move from 6-bit to 8-bit; all other layer assignments are identical.

⚠️ Honest assessment: While this variant ties for best on MMLU_PRO (82.0%) — confirming the 8-bit routed experts improve knowledge recall — it underperforms the 6-bit XL on math reasoning (see Custom MATHQA below). On the failure-enriched MATHQA test, the 8-bit XL scored 56.7% vs the 6-bit XL's 66.7%, and produced the highest average response length (8,789 chars vs 5,894). For reasoning-focused workloads, the 6-bit XL remains the recommended XL variant. This model is the knowledge specialist — use it when broad factual recall matters more than step-by-step math.

Note: tie_word_embeddings=True means there is no separate lm_head (tied to embed_tokens).

Gemma-4-26B-A4B is a 25.2B-parameter MoE (Mixture of Experts) model with 128 experts per layer (avg. 3.8B active parameters per token) and a vision encoder.

About XL Quantization

BaseQuant_XL is a fully data-agnostic, static quantization. No calibration dataset, no sensitivity analysis, no importance matrix. Precision is allocated purely by architectural role — routing-critical layers get higher precision, bulk expert parameters get lower precision. The result is a transparent, faithful capture of the source model.

Data-dependent calibration quantizations (iMatrix, AWQ, GPTQ, oQ, oQ4e, etc.) use a calibration set to guide bit allocation. This can produce a skewed representation of the model: domains well-represented in the calibration data (English, popular topics, public or leaked benchmarks) are preserved better, while underrepresented domains (non-English languages, niche use cases, your own data) are preserved worse. XL avoids this trade-off entirely — it generalizes honestly because it is never fit to any particular data distribution.

Intelligence Benchmarks

All variants of google/gemma-4-26B-A4B-it quantized to MLX, run under identical conditions on the same hardware.

Instruct-mode benchmarks (no thinking)

Custom MATHQA (thinking mode, relaxed 8192-token budget)

A failure-enriched MATHQA test set (n=30) run with thinking enabled and relaxed token limits to isolate reasoning quality from throughput artifacts.

⚠️ Dataset quality caveat: This benchmark was run with the failure-enriched MATHQA test set before corrections. A full re-audit (2026-07-21) found 9 of 19 "hard" questions had corrupted text, wrong expected answers, or unsolvable constraints (e.g., dropped digits, wrong combinatorial formulas, P(A∩B)>P(A)). All 9 have since been corrected in gemma-4-results/test_sets.json — the test set is now mathematically sound. The scores below were obtained with the uncorrected set and carry this caveat. See the 6-bit XL model card for methodology details.

Use with mlx

pip install -U mlx-vlm

python -m mlx_vlm.generate --model leonsarmiento/gemma-4-26B-A4B-it-8bit-XL-mlx --max-tokens 256 --temperature 1.0 --top-p 1.0 --prompt "Hello"

BaseQuant_XL Quantization Strategy

Quantization Details

• Quantization type: BaseQuant_XL mixed (multimodal, vision preserved)
• Bits per weight: 8.833
• Total size: ~27 GB (6 shards)
• Group size: 64
• Method: Custom quant_predicate via mlx_vlm
• Note: tie_word_embeddings=True — no separate lm_head

Recommended Inference Parameters

Chat Template

The chat template is synced to the canonical Google upstream version from google/gemma-4-26B-A4B-it/chat_template.jinja (last verified 2025-07-09). Do not rely on the HF cache copy — it may be stale.

Key markers in the latest template: preserve_thinking, continues_into_next, prev_non_tool_role, null handling in format_argument.

The template is injected into both chat_template.jinja (standalone file) and tokenizer_config.json (chat_template JSON field).