Downloads · 30 days
27
7% of all-time downloads
Brooooooklyn/Gemma-4-26B-A4B-IT-UD-MXFP8_K_XL-mlx
Gemma-4-26B-A4B-IT-UD-MXFP8_K_XL-mlx is a text generation model from Brooooooklyn. Use it when you need the model to write or continue text. It is set up for mlx-node. The card lists the license as gemma.
MXFP8 (OCP micro-scaling FP8) quantization of google/gemma-4-26b-a4b-it for Apple Silicon, using the Unsloth Dynamic quantization strategy via mlx-node.
Downloads · 30 days
27
7% of all-time downloads
All-time downloads
385
Public
Parameters
25.8B
27.6 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors27.6 GB · 100%
How the weights are stored.
U3224.8B · 96%
From the Hugging Face model README
MXFP8 (OCP micro-scaling FP8) quantization of google/gemma-4-26b-a4b-it for Apple Silicon, using the Unsloth Dynamic quantization strategy via mlx-node.
| Original (BF16) | UD-MXFP8_K_XL (this model) | |
|---|---|---|
| Size | ~49 GB | 26 GB |
| Format | SafeTensors | SafeTensors |
| Precision | BF16 uniform | MXFP8 (E8M0 scales) + BF16 |
| FFN group size | — | 32 |
| Biases | — | no |
MXFP8 is the Open Compute Project (OCP) micro-scaling FP8 format. Each group of 32 elements shares a single 8-bit E8M0 scale (a power-of-two exponent), and elements themselves are stored as E4M3 FP8 values. Compared to 8-bit affine:
For typical LLM weight distributions, MXFP8 retains quality on par with 8-bit affine while shrinking the metadata footprint.
| Repo | Bit budget | Size | Decode (tok/s) |
|---|---|---|---|
| Brooooooklyn/Gemma-4-26B-A4B-IT-UD-Q3_K_XL-mlx | 3-bit base | 14 GB | 60.6 |
| Brooooooklyn/Gemma-4-26B-A4B-IT-UD-MXFP4_K_XL-mlx | mxfp4 | 16 GB | 58.4 |
| Brooooooklyn/Gemma-4-26B-A4B-IT-UD-Q4_K_XL-mlx | 4-bit base | 17 GB | 58.6 |
| Brooooooklyn/Gemma-4-26B-A4B-IT-UD-NVFP4_K_XL-mlx | nvfp4 | 17 GB | 57.9 |
| Brooooooklyn/Gemma-4-26B-A4B-IT-UD-Q5_K_XL-mlx | 5-bit base | 20 GB | 50.3 |
| Brooooooklyn/Gemma-4-26B-A4B-IT-UD-Q6_K_XL-mlx | 6-bit base | 23 GB | 51.9 |
| Brooooooklyn/Gemma-4-26B-A4B-IT-UD-MXFP8_K_XL-mlx (this model) | mxfp8 | 26 GB | 49.8 |
| Brooooooklyn/Gemma-4-26B-A4B-IT-UD-Q8_K_XL-mlx | 8-bit base | 27 GB | 49.8 |
Benchmarked on Apple M3 Max 128GB via examples/lm.ts (best decode tok/s across turns 2–4, steady-state, capitals chat with reasoningEffort: 'low').
Note: No Q2 variant is published — Gemma-4-26B-A4B-IT has only ~4B active parameters per token, which is below the architectural redundancy needed for 2-bit quantization to remain coherent. Both unsloth and mixed_2_6 recipes produced gibberish at Q2 on this model.
Steady-state decode: 49.8 tok/s on Apple M3 Max 128GB (best of turns 2–4, examples/lm.ts capitals chat with reasoningEffort: 'low'). Decode is memory-bandwidth bound on Apple Silicon — fewer bytes per token directly translates to higher throughput. The MoE architecture activates only top-K of 128 experts per token (~4B active out of ~26B total), and the compiled C++ forward graph fuses the per-layer dispatch.
| Weight | Mode | Bits | Group | Rationale |
|---|---|---|---|---|
self_attn.q_proj | mxfp8 | 8 | 32 | AWQ-corrected via input_layernorm |
self_attn.k_proj | mxfp8 | 8 | 32 | AWQ-corrected via input_layernorm |
self_attn.v_proj | mxfp8 | 8 | 32 | AWQ-corrected via input_layernorm (only on full-attention layers) |
mlp.gate_proj | mxfp8 | 8 | 32 | Shared dense MLP |
mlp.up_proj | mxfp8 | 8 | 32 | Shared dense MLP |
mlp.down_proj | mxfp8 | 8 | 32 | Shared dense MLP; "slightly more sensitive" (unsloth base+1) |
experts.switch_glu.gate_proj | mxfp8 | 8 | 32 | MoE expert gate (per-expert across all 128); base bits |
experts.switch_glu.up_proj | mxfp8 | 8 | 32 | MoE expert up (per-expert across all 128); base bits |
experts.switch_glu.down_proj | mxfp8 | 8 | 32 | MoE expert down (per-expert across all 128 + routing); unsloth base+1 |
self_attn.o_proj | bf16 | — | — | NOT AWQ-correctable; kept full-precision |
Built on Unsloth Dynamic 2.0 per-tensor KLD analysis. At --q-bits 8 the unsloth recipe's per-layer bit offsets all snap to 8-bit. Then --q-mxfp orthogonally promotes every 8-bit affine decision to MXFP8 (mode="mxfp8", bits=8, group_size=32) — except for keys whose dequantizers are affine-only (embed_tokens, lm_head, router.proj, embed_vision.embedding_projection).
imatrix AWQ pre-scaling amplifies important weight channels and fuses inverse scales into preceding layer norms (zero inference overhead). At 8-bit, the recipe's primary contribution is the affine-only safety net for embedding and routing layers.
| Parameter | Value |
|---|---|
| Total parameters | ~26B (~4B active per token) |
| Hidden size | 2,816 |
| Layers | 30 (sliding-window attention) |
| Attention heads | 16 (8 KV heads, GQA 2:1) |
| Head dimension | 256 |
| Experts | 128 per MoE layer |
| MoE intermediate size | 704 |
| Vocab size | 262,144 |
| Max context | 262,144 tokens |
| Vision | yes (Gemma4ForConditionalGeneration) |
import { loadSession } from '@mlx-node/lm';
const session = await loadSession('./Gemma-4-26B-A4B-IT-UD-MXFP8_K_XL-mlx');
for await (const event of session.sendStream('Explain the MoE architecture in Gemma-4.', {
config: { maxNewTokens: 2048, temperature: 0.6, reasoningEffort: 'low' },
})) {
if (!event.done) process.stdout.write(event.text);
}
mlx convert \
-i gemma-4-26b-a4b-it \
-o Gemma-4-26B-A4B-IT-UD-MXFP8_K_XL-mlx \
-q --q-bits 8 --q-mxfp --q-recipe unsloth \
--imatrix-path imatrix_unsloth.gguf
Gemma Terms of Use (inherited from base model).