gemma-4-26b-a4b-mlx-imatrix3plus-awq-high6-top40

gemma-4-26b-a4b-mlx-imatrix3plus-awq-high6-top40

google/gemma-4-26B-A4B-it quantized to 5.057 bpw for Apple Silicon (MLX) — the Korean-chat flagship of the lumen Gemma 4 family. Built by combining HIGH=6 (top sensitivity tier → 6-bit) with top4_fraction=0.40 (40 % of tensors elevated to top tier instead of the default 35 %).

Positioning in the 3-tier family

External benchmark context

This is the lowest-PPL MLX-loadable Gemma 4 26B-A4B quantization currently available:

NVIDIA NVFP4 (TensorRT-only) and cyankiwi/...-AWQ-4bit (compressed-tensors, vLLM-only) formats are not MLX-loadable and therefore not directly comparable on Apple Silicon.

Multi-angle eval (vs ship & Quality tier)

When to choose this tier: Korean conversational workloads, instruction-following on tulu-like calibration distributions, long-form Korean generation. It maximises HAERAE and tulu PPL at the cost of broad-MMLU and English-CoT-math (where ship/HIGH=6 remain better).

Why top4_fraction=0.40 on HIGH=6?

After confirming HIGH=6 as the optimal top tier (−4.18 ppl), we re-swept top4_fraction under the new tier. Result: sharp non-monotonic peak at 0.40:

The 35 → 40 % threshold contains a discrete tensor cluster that delivers a quality jump independent of the bit tier itself. AWQ re-search at this top4 setting gave 58/60 bit-identical scales to ship → confirms AWQ-recipe coupling is robust to ±10 % top4 shift on the HIGH=6 base.

Usage (MLX)

pip install -U mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("hsng95/gemma-4-26b-a4b-mlx-imatrix3plus-awq-high6-top40")

messages = [
    {"role": "system", "content": "당신은 도움이 되는 한국어 비서입니다."},
    {"role": "user", "content": "양자화가 무엇인지 한 문단으로 설명해줘."},
]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
print(generate(model, tokenizer, prompt=prompt, max_tokens=512))

⚠️ Gemma 4 26B-A4B is instruction-tuned. Always use apply_chat_template — raw prompts produce garbage on every Gemma 4 quant (3-bit, 4-bit, AWQ alike).

Sampling

Per the upstream Gemma 4 recommendation:

• temperature=1.0
• top_p=0.95
• top_k=64

Quantization details

• Source: google/gemma-4-26B-A4B-it (bf16)
• imatrix corpus: 38 sequences, ~68 % Korean / 37 % conversational (lumen imatrix3plus)
• AWQ: Option B (mean_sq proxy), mlp_down absorption groups filtered out, embed_tokens at AFFINE 8-bit
• Bit allocation: top 40 % sensitivity → 6-bit AFFINE, middle 25 % → 4-bit, bottom 35 % → 3-bit
• Group size: 64
• Build tool: lumen-rs/scripts/quant/streaming_awq_quantize_strict.py --high-bits 6 --top4-fraction 0.40

Acknowledgments

Built on top of mlx-community/gemma-4-26b-a4b-4bit (DWQ source) for imatrix capture, and the upstream google/gemma-4-26B-A4B-it for bf16 source weights.

License

Apache 2.0 (Gemma 4 license terms apply, see Gemma license).