Downloads · 30 days
30
8% of all-time downloads
leonsarmiento/Huihui-gemma-4-26B-A4B-it-8bit-XL-mlx
Huihui-gemma-4-26B-A4B-it-8bit-XL-mlx is a image-text-to-text model from leonsarmiento. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for mlx.
This model was converted to MLX format from huihui-ai/Huihui-gemma-4-26B-A4B-it-abliterated using BaseQuantXL 8-bit mixed quantization — the maximum-fidelity XL variant. The vision encoder is preserved (full multimodal).
Downloads · 30 days
30
8% of all-time downloads
All-time downloads
360
Public
Parameters
25.8B
28.5 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors28.5 GB · 100%
How the weights are stored.
U3224.7B · 96%
From the Hugging Face model README
This model was converted to MLX format from huihui-ai/Huihui-gemma-4-26B-A4B-it-abliterated using BaseQuant_XL 8-bit mixed quantization — the maximum-fidelity XL variant. The vision encoder is preserved (full multimodal).
BaseQuant_XL keeps the most routing-critical layers in full bf16 precision — the MoE router.proj (gate) and the mlp (shared/dense expert that processes all tokens) — while applying 8-bit quantization to everything else. Compared to the 6-bit XL variant, only the routed experts (experts.switch_glu) move from 6-bit to 8-bit; all other layer assignments are identical.
This is the abliterated (uncensored) version of Gemma-4-26B-A4B, a 25.2B-parameter MoE (Mixture of Experts) model with 128 experts per layer (avg. 3.8B active parameters per token) and a vision encoder. The unquantized BF16 base was used as the source for maximum fidelity.
Note: tie_word_embeddings=True means there is no separate lm_head (tied to embed_tokens).
BaseQuant_XL is a fully data-agnostic, static quantization. No calibration dataset, no sensitivity analysis, no importance matrix. Precision is allocated purely by architectural role — routing-critical layers get higher precision, bulk expert parameters get lower precision. The result is a transparent, faithful capture of the source model.
Data-dependent calibration quantizations (iMatrix, AWQ, GPTQ, oQ, oQ4e, etc.) use a calibration set to guide bit allocation. This can produce a skewed representation of the model: domains well-represented in the calibration data (English, popular topics, public or leaked benchmarks) are preserved better, while underrepresented domains (non-English languages, niche use cases, your own data) are preserved worse. XL avoids this trade-off entirely — it generalizes honestly because it is never fit to any particular data distribution.
pip install -U mlx-vlm
python -m mlx_vlm.generate --model leonsarmiento/Huihui-gemma-4-26B-A4B-it-8bit-XL-mlx --max-tokens 256 --temperature 1.0 --top-p 1.0 --prompt "Hello"
| Bit Depth | Layers | Rationale |
|---|---|---|
| bf16 (unquantized) | router.proj (gate), mlp (shared/dense expert), vision tower | Routing decisions and the dense expert that processes every token — errors here cascade through all tokens |
| 8-bit | embed_tokens, self_attn, experts.switch_glu (routed experts) | Near-lossless — the max-fidelity XL recipe. Only the routing-critical bf16 paths are kept at full precision |
| Layer | Bits | Group Size |
|---|---|---|
router.proj (gate) | bf16 | — |
mlp (shared/dense expert) | bf16 | — |
| Vision tower (full) | bf16 | — |
embed_tokens | 8 | 64 |
self_attn | 8 | 64 |
experts.switch_glu (routed experts) | 8 | 64 |
| Default fallback | 8 | 64 |
quant_predicate via mlx_vlmtie_word_embeddings=True — no separate lm_head| Parameter | Value |
|---|---|
temperature | 1.0 |
top_p | 1.0 |
top_k | 64 |
min_p | 0.01 |
repeat_penalty | 1.05 |
<|channel|>thought<|channel|>The chat template is synced to the canonical Google upstream version from google/gemma-4-26B-A4B-it/chat_template.jinja. Do not rely on the HF cache copy — it may be stale.
Key markers in the latest template: preserve_thinking, continues_into_next, prev_non_tool_role, null handling in format_argument.
The template is injected into both chat_template.jinja (standalone file) and tokenizer_config.json (chat_template JSON field).