Downloads · 30 days
93
26% of all-time downloads
roman220220/gemma-4-26B-A4B-it-gptq-mlx-jang
gemma-4-26B-A4B-it-gptq-mlx-jang is a image-text-to-text model from roman220220. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for mlx. The card lists the license as apache-2.0.
<p align="center" <img src="llmtray-banner.png" alt="LLMTray" width="100%" </p
Downloads · 30 days
93
26% of all-time downloads
All-time downloads
353
Public
Parameters
27.8B
15.6 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors15.6 GB · 100%
How the weights are stored.
U3227.8B · 100%
From the Hugging Face model README
▶ Run it locally in LLMTray
A free, native macOS app for local AI on Apple Silicon — chat, images, music, agents and an OpenAI-compatible API. Point it at this model; nothing leaves your Mac.
A JANG-style mixed-precision MLX quantization of google/gemma-4-26B-A4B-it — the 26.5B-parameter, 128-expert / top-8 MoE variant of Gemma 4 (≈4B active parameters per token, hence "A4B") — using GPTQ Hessian-based error correction across every quantizable Linear in both the text decoder and the vision tower. Attention projections at 8-bit, feed-forward (dense MLP and all 128 routed experts) at 4-bit, embeddings at 8-bit RTN. Nothing is left in bf16.
audio_config is null on the real
checkpoint) — text + vision only, unlike the E4B sibling.| Role | Bits | Notes |
|---|---|---|
Attention (q/k/v/o_proj, text + vision) | 8-bit GPTQ | Small, error-sensitive |
Dense MLP (gate/up/down_proj) | 4-bit GPTQ | Every layer has a dense MLP and MoE experts (hybrid) |
MoE experts (experts.gate_up_proj/down_proj, 128 per layer) | 4-bit GPTQ | The bulk of the parameters; calibrated per-expert on the tokens actually routed to each |
Router (router.*) | untouched | Routing precision is disproportionately sensitive — never quantized |
Embeddings (embed_tokens) | 8-bit RTN | No shared-input matmul for GPTQ to correct |
Calibrated with --group-size 64 on real data: diverse text prompts for
the language model, real COCO photographs for the vision tower.
hidden_states_1 + hidden_states_2) — the router operates on the
pre-dense-MLP residual. Both paths are quantized here.Gemma4TextExperts module consumes
raw stacked [128, out, in] nn.Parameter tensors inside a Python
loop, not nn.Linear submodules — so calibration hooks the whole
experts module, replicates its real expert_mask/token_idx gathering
to bucket calibration rows per expert, and runs batched GPTQ across all
128 experts at once. down_proj's calibration input is recomputed as
act_fn(gate)*up using the real (uncorrected) gate_up_proj, exactly
as the real forward pass feeds it.mx.quantize supports
(32/64/128 — 4304 = 2⁴·269). Rather than leave those tensors in bf16,
gate/up/down_proj are zero-padded to 4352 (next multiple of 64) — this
is mathematically exact (GELU(0)·0 = 0, and the padded activations hit
correspondingly-zero-weighted down_proj columns), declared in
config.json's vision_config.intermediate_size so the model is built
at the padded width with no inference-code changes.Text: correctly answers factual/reasoning prompts (Gemma 4's thinking-mode chain included).
Vision (real COCO photo of two cats, "What animal is in this picture?"):
There are two cats in this picture.
Both run through this actual GPTQ-quantized checkpoint (MoE routing + vision tower + fusion all exercised).
pip install git+https://github.com/ipsupport-llc/mlx-lm.git@main
from mlx_lm import load, generate
model, tok = load("roman220220/gemma-4-26B-A4B-it-gptq-mlx-jang")
messages = [{"role": "user", "content": "What is the capital of France?"}]
print(generate(model, tok, prompt=tok.apply_chat_template(messages, add_generation_prompt=True)))
Image input needs a manual generation loop (mlx_lm.generate() has no
image plumbing for this fork's vision models yet) — full example in the
gemma4-quant pipeline repo.
gemma4.py, gemma4_text.py, gemma4_vision.py)gemma4_26b_gptq_calibrate.py, gemma4_26b_gptq_splice.py)gptq_nbit/gptq_nbit_batched implementation as this account's other quantization work (nemotron-extreme-quant, zimage-quant), unmodified.Local-first AI tools for macOS by IPSupport — nothing leaves your Mac.
Runs in LLMTray as a chat model, with vision.
Licensed under the Apache License 2.0, the same license as the base model — see LICENSE.
Modified from google/gemma-4-26B-A4B-it: the text decoder and vision tower were GPTQ-quantized (attention 8-bit; dense MLP and all 128 MoE experts 4-bit; group size 64), embeddings were quantized to 8-bit RTN, the router was left untouched, the vision MLP was zero-padded from 4304 to 4352 (with config.json updated to match), and the model was converted to MLX. The weights and configuration files in this repo are therefore modified versions of the original, not the original files.
Gemma 4 is released by Google under Apache 2.0 (Gemma 4 license terms).