gemma-4-12B-it-6bit-XL-mlx

gemma-4-12B-it-6bit-XL-mlx

MLX BaseQuant_XL 6/8-bit mixed quantization of google/gemma-4-12B-it for Apple Silicon.

BaseQuant_XL Strategy

This is the dense XL recipe — precision allocated by layer criticality, not by parameter count.

Why no bf16 MLP?

In the MoE XL recipe, the shared/dense MLP stays bf16 because routed experts (6-bit) carry the bulk and are sparsely activated. In a dense model, the MLP is the bulk — keeping it at bf16 would negate the purpose of quantization. Instead, we elevate attention from 6-bit to 8-bit (vs the older _68 recipe that quantized both at 6-bit), preserving the component most sensitive to quantization noise.

Quantization Details

• Quantization type: BaseQuant_XL mixed (multimodal, vision+audio preserved)
• Method: Custom quant_predicate via mlx_vlm
• Group size: 64

About XL Quantization

BaseQuant_XL is a fully data-agnostic, static quantization. No calibration dataset, no sensitivity analysis, no importance matrix. Precision is allocated purely by architectural role — critical layers get higher precision, bulk parameters get lower precision. The result is a transparent, faithful capture of the source model.

Data-dependent calibration quantizations (iMatrix, AWQ, GPTQ, oQ, oQ4e, etc.) use a calibration set to guide bit allocation. This can produce a skewed representation of the model: domains well-represented in the calibration data (English, popular topics, public or leaked benchmarks) are preserved better, while underrepresented domains (non-English languages, niche use cases, your own data) are preserved worse. XL avoids this trade-off entirely — it generalizes honestly because it is never fit to any particular data distribution.

Recommended Inference Parameters

Reasoning Parsing in LM Studio and Others

• Start string: <|channel|>thought
• End string: <|channel|>

Usage

pip install -U mlx-vlm

python -m mlx_vlm.generate --model leonsarmiento/gemma-4-12B-it-6bit-XL-mlx --max-tokens 256 --temperature 1.0 --top-p 1.0 --prompt "Hello"

Intelligence Benchmarks (Instruct Mode, n=50)

XL beats data-aware OptiQ 4-bit across the board (+4 MMLU, +2 MMLU_PRO, +2 ARC, +4 HUMANEVAL, +2.4 avg), while the 26B MoE big sister retains the crown as expected (+6 points avg, but 2.1× the size).

⚠️ n=50 sampling means wide confidence intervals (±~13% at 95% CI). Differences under ~6 points may not be statistically significant. Data-aware methods (OptiQ) may be calibrated on benchmark-like data — their scores carry a benchmaxxing caveat. BaseQuant_XL (data-agnostic) provides the most honest generalization estimate.

---

Local SOTA Comparison (for context)

The 26B XL is the current champion for 48GB Macs. The dense 12B offers nearly identical architecture at half the size:

Chat Template

Ships with Google's canonical chat_template.jinja (18,681 chars) in both chat_template.jinja and tokenizer_config.json. Includes preserve_thinking support for tool-calling workflows.