gemma-4-26b-a4b-mlx-imatrix3plus-awq-high6

gemma-4-26b-a4b-mlx-imatrix3plus-awq-high6

google/gemma-4-26B-A4B-it quantized to 4.674 bpw for Apple Silicon (MLX). Built by elevating the top sensitivity tier from 4-bit to 6-bit AFFINE on top of the ship recipe (hsng95/gemma-4-26b-a4b-mlx-imatrix3plus-awq).

Positioning in the 3-tier family

Why HIGH=6?

The top sensitivity-ranked 35 % of tensors get elevated from 4-bit to 6-bit AFFINE. At 6-bit, quantization noise becomes small enough that the AWQ rescaling baked in from the 4-bit-noise calibration becomes a near-no-op rather than a mismatch — yielding pure precision gain instead of calibration drift.

Empirically the bit-tier ridge is non-monotonic and HIGH=6 sits on the optimum:

Multi-angle eval (vs ship)

Tested across 11 measurement axes including downstream tasks:

Caveat: GSM8K shows a sharp regression on no-shot chain-of-thought generation despite winning multi-choice MMLU/ARC. If you primarily target math with no-shot CoT, use the ship tier. For balanced knowledge + Korean + English chat, this tier is the best default.

Usage (MLX)

pip install -U mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("hsng95/gemma-4-26b-a4b-mlx-imatrix3plus-awq-high6")

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Write a short joke about saving RAM."},
]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
print(generate(model, tokenizer, prompt=prompt, max_tokens=256))

⚠️ Gemma 4 26B-A4B is instruction-tuned. Always use apply_chat_template — raw prompts produce garbage on every Gemma 4 quant (3-bit, 4-bit, AWQ alike).

Sampling

Per the upstream Gemma 4 recommendation:

• temperature=1.0
• top_p=0.95
• top_k=64

Quantization details

• Source: google/gemma-4-26B-A4B-it (bf16)
• imatrix corpus: 38 sequences, ~68 % Korean / 37 % conversational (lumen imatrix3plus)
• AWQ: Option B (mean_sq proxy), mlp_down absorption groups filtered out, embed_tokens at AFFINE 8-bit
• Bit allocation: top 35 % sensitivity → 6-bit AFFINE, middle 30 % → 4-bit, bottom 35 % → 3-bit
• Group size: 64
• Build tool: lumen-rs/scripts/quant/streaming_awq_quantize_strict.py --high-bits 6

Acknowledgments

Built on top of mlx-community/gemma-4-26b-a4b-4bit (DWQ source) for imatrix capture, and the upstream google/gemma-4-26B-A4B-it for bf16 source weights.

License

Apache 2.0 (Gemma 4 license terms apply, see Gemma license).