gemma-4-26B-A4B-it-6bit-XL-mlx

Flight Simulator — Coded by This Model

A real-world coding benchmark: each model was prompted to write a complete flight simulator from scratch. The resulting code was rendered and recorded.

About XL Quantization

BaseQuant_XL is a fully data-agnostic, static quantization. No calibration dataset, no sensitivity analysis, no importance matrix. Precision is allocated purely by architectural role — routing-critical layers get higher precision, bulk expert parameters get lower precision. The result is a transparent, faithful capture of the source model.

Data-dependent calibration quantizations (iMatrix, AWQ, GPTQ, oQ, oQ4e, etc.) use a calibration set to guide bit allocation. This can produce a skewed representation of the model: domains well-represented in the calibration data (English, popular topics, public or leaked benchmarks) are preserved better, while underrepresented domains (non-English languages, niche use cases, your own data) are preserved worse. XL avoids this trade-off entirely — it generalizes honestly because it is never fit to any particular data distribution.

Thinking-Mode MATHQA: Gemma vs Qwen3.6 Token Efficiency

Both models were tested on the corrected custom MATHQA test set (n=30, thinking mode, 8192-token budget). Both verified at 30/30 (100%) after re-test of failures. The decisive difference is how efficiently they arrive at correct answers.

Gemma uses 29% fewer reasoning chars per correct answer than Qwen3.6 (verified results after re-test of failures). Gemma's shorter reasoning chains mean:

• Less context window consumed — more space for multi-turn conversation or complex prompts
• Less energy per answer — fewer GPU cycles wasted on self-verification
• More decisive reasoning — Gemma commits to answers sooner; Qwen3.6 tends to verify and re-derive

The per-question scatter shows Gemma's reasoning clustered at 1K–4K chars while Qwen3.6 spreads from 2K–12K. On the hardest questions (Q2232, Q2932), both models spike to 11K–19K chars — the efficiency gap closes when the problem genuinely requires extended reasoning.

⚠️ Raw accuracy differences are dominated by sampling variance (temp=1.0) and answer-extraction artifacts. Both models are capable of 30/30. The real differentiator is reasoning efficiency — correct answers per thousand reasoning chars.

Local SOTA for 48GB Macs — Intelligence Benchmark Comparison

This model is part of a benchmark comparison of the best local MLX-quantized LLMs that fit in 48GB unified memory on Apple Silicon. All benchmarks run in instruct mode (no thinking) with n=50 samples per benchmark.

Collection: Local SOTA for 48GB Macs

⚠️ n=50 sampling means wide confidence intervals (±~13% at 95% CI). Differences under ~6 points may not be statistically significant. Models using data-aware quantization (oQ/oQe) may be calibrated on benchmark-like data — their scores carry a benchmaxxing caveat. The BaseQuant_XL variants (data-agnostic) provide the most honest generalization estimates.

leonsarmiento/gemma-4-26B-A4B-it-6bit-XL-mlx

This model was converted to MLX format from google/gemma-4-26B-A4B-it using BaseQuant_XL 6/8-bit mixed quantization optimized for Apple Silicon. The vision encoder is preserved and quantized at 6-bit, making this a full multimodal model.

BaseQuant_XL keeps the most routing-critical layers in full bf16 precision — the MoE router.proj (gate) and the mlp (shared/dense expert that processes all tokens) — while applying aggressive quantization to the bulk parameters. Note: tie_word_embeddings=True means there is no separate lm_head (tied to embed_tokens).

Gemma-4-26B-A4B is a 25.2B-parameter MoE (Mixture of Experts) model with 128 experts per layer (avg. 3.8B active parameters per token) and a vision encoder.

Chat Template

The chat_template.jinja and tokenizer_config.json were synced to the latest canonical Gemma 4 chat template from Google (published 2025-07-09) at quantization time. This update fixes tool-calling loops, turn closures, and thinking content-ordering issues. See google/gemma-4-26B-A4B-it/chat_template.jinja for the upstream source.

Quantization Details

• Quantization type: BaseQuant_XL mixed (multimodal, vision preserved)
• Bits per weight: 7.061
• Total size: ~21 GB (5 shards)
• Group size: 64
• Method: Custom quant_predicate via mlx_vlm
• Note: tie_word_embeddings=True — no separate lm_head

BaseQuant_XL Strategy

Intelligence Benchmarks

¹ Custom relaxed MATHQA test (see methodology below). Standard benchmark MATHQA/GSM8K results were omitted because time-constrained math benchmarks conflate throughput with reasoning quality — see Curated MATHQA Methodology.

Analysis: XL 6/8 at 21GB matches or beats the 8-bit uniform (26GB) on knowledge and code-MC benchmarks — MMLU_PRO (+2 over 6-bit), MBPP (+3.3 over 6-bit), LIVECODEBENCH (+6.7 over 8-bit). On the custom MATHQA test with relaxed token budget, the XL trails only the QAT 4-bit and beats both uniform variants.

Custom MATHQA Methodology

Standard math benchmarks (run via LM Studio's evaluation harness with fixed time/token budgets) produced results that conflated generation throughput with reasoning quality. The XL's bf16 router/shared-expert layers make it 1.2x–5.5x slower per answer than the uniform variants, causing it to exhaust the benchmark budget on harder problems — 100% of the XL's failures were timeouts, with zero arithmetic, logic, or formatting errors.

To get a fairer picture of reasoning quality, a custom curated MATHQA test was constructed:

Test Set Construction (n=30)

1. Failure-enriched sample: All 19 MATHQA questions where either the XL or the 6-bit uniform produced an empty/timeout response in the original benchmark — these are the "hard" cases where time limits matter most.
2. Control group: 11 questions both models solved correctly in the original run, sampled randomly. If a model starts missing these, it indicates a real reasoning issue.
3. Relaxed constraints: 8192 max tokens (vs. the original ~2048 thinking limit), no server-side timeout. Each model generates as long as it needs.
4. All 4 variants tested: QAT 4-bit, 6-bit uniform, 8-bit uniform, XL 6/8 — same questions, same prompt, same parameters (temperature 1.0, top_p 0.95).

Detailed Results

All 11 control-group questions were answered correctly by all 4 models — none of the variants exhibit a real reasoning deficit on solvable problems.

Recommended Inference Parameters

Reasoning Parsing in LM Studio and Others

• Start string: <|channel>thought
• End string: <channel|>

Use with mlx

pip install -U mlx-vlm

python -m mlx_vlm.generate --model leonsarmiento/gemma-4-26B-A4B-it-6bit-XL-mlx --max-tokens 256 --temperature 1.0 --top-p 1.0 --prompt "Hello"