Downloads · 30 days
16
4% of all-time downloads
Brooooooklyn/Gemma-4-31B-IT-UD-NVFP4_K_XL-mlx
Gemma-4-31B-IT-UD-NVFP4_K_XL-mlx is a text generation model from Brooooooklyn. Use it when you need the model to write or continue text. It is set up for mlx-node. The card lists the license as gemma.
NVFP4 (NVIDIA Blackwell FP4) quantization of google/gemma-4-31b-it for Apple Silicon, using the Unsloth Dynamic quantization strategy via mlx-node.
Downloads · 30 days
16
4% of all-time downloads
All-time downloads
389
Public
Parameters
31.3B
25.4 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors25.4 GB · 100%
How the weights are stored.
U3227.6B · 88%
From the Hugging Face model README
NVFP4 (NVIDIA Blackwell FP4) quantization of google/gemma-4-31b-it for Apple Silicon, using the Unsloth Dynamic quantization strategy via mlx-node.
| Original (BF16) | UD-NVFP4_K_XL (this model) | |
|---|---|---|
| Size | ~62 GB | 24 GB |
| Format | SafeTensors | SafeTensors |
| Precision | BF16 uniform | NVFP4 (E4M3 scales) + mixed affine + BF16 |
| FFN group size | — | 16 |
| Biases | — | no (MLP); yes (affine layers) |
NVFP4 is NVIDIA's Blackwell FP4 micro-scaling format. Each group of 16 elements shares a single 8-bit E4M3 scale (a full FP8 number with mantissa, not just an exponent), and elements themselves are stored as E2M1 FP4 values. Compared to MXFP4 (OCP):
For dense LLM weights, NVFP4 typically beats MXFP4 on perplexity at the same nominal bit budget, with the trade-off of slightly more metadata per weight.
| Repo | Bit budget | Size | Decode (tok/s) |
|---|---|---|---|
| Brooooooklyn/Gemma-4-31B-IT-UD-Q2_K_XL-mlx | 2-bit base | 18 GB | 12.1 |
| Brooooooklyn/Gemma-4-31B-IT-UD-Q3_K_XL-mlx | 3-bit base | 21 GB | 10.4 |
| Brooooooklyn/Gemma-4-31B-IT-UD-Q4_K_XL-mlx | 4-bit base | 24 GB | 9.2 |
| Brooooooklyn/Gemma-4-31B-IT-UD-MXFP4_K_XL-mlx | mxfp4 | 24 GB | 10.6 |
| Brooooooklyn/Gemma-4-31B-IT-UD-NVFP4_K_XL-mlx (this model) | nvfp4 | 24 GB | 9.9 |
| Brooooooklyn/Gemma-4-31B-IT-UD-Q5_K_XL-mlx | 5-bit base | 28 GB | 6.8 |
| Brooooooklyn/Gemma-4-31B-IT-UD-Q6_K_XL-mlx | 6-bit base | 31 GB | 7.2 |
| Brooooooklyn/Gemma-4-31B-IT-UD-MXFP8_K_XL-mlx | mxfp8 | 34 GB | 7.8 |
| Brooooooklyn/Gemma-4-31B-IT-UD-Q8_K_XL-mlx | 8-bit base | 35 GB | 7.2 |
Benchmarked on Apple M3 Max 128GB via examples/lm.ts (best decode tok/s across turns 2–4, steady-state, capitals chat with reasoningEffort: 'low').
Steady-state decode: 9.9 tok/s on Apple M3 Max 128GB (best of turns 2–4, examples/lm.ts capitals chat with reasoningEffort: 'low'). Decode is memory-bandwidth bound on Apple Silicon — fewer bytes per token directly translates to higher throughput. Gemma-4-31B is fully dense (all 31B parameters active per token), so each decoded token must stream the entire quantized weight footprint from unified memory.
| Weight | Mode | Bits | Group | Rationale |
|---|---|---|---|---|
embed_tokens | 6-bit affine | 6 | 64 | Tied with lm_head (Gemma4 shares weights); affine-only loader |
self_attn.q_proj | 6-bit affine | 6 | 64 | AWQ-corrected via input_layernorm |
self_attn.k_proj | 6-bit affine | 6 | 64 | AWQ-corrected via input_layernorm |
self_attn.v_proj | 6-bit affine | 6 | 64 | AWQ-corrected via input_layernorm |
mlp.gate_proj | nvfp4 | 4 | 16 | Dense MLP (top-level default) (top-level default) |
mlp.up_proj | nvfp4 | 4 | 16 | Dense MLP (top-level default) (top-level default) |
mlp.down_proj | 5-bit affine | 5 | 64 | Dense MLP; "slightly more sensitive" (unsloth base+1) |
embed_vision.embedding_projection | 8-bit affine | 8 | 64 | Vision adapter; affine-only loader |
self_attn.o_proj | bf16 | — | — | NOT AWQ-correctable; kept full-precision |
Built on Unsloth Dynamic 2.0 per-tensor KLD analysis. At --q-bits 4 (implicit under --q-mode nvfp4) the unsloth recipe assigns 4-bit to MLP gate/up projections, 5-bit affine to down_proj, 6-bit affine + AWQ pre-scaling to attention q/k/v, and keeps self_attn.o_proj as bf16. Then --q-mode nvfp4 orthogonally promotes only the 4-bit affine decisions to NVFP4 (mode="nvfp4", bits=4, group_size=16) — non-4-bit decisions stay affine, and affine-only loaders (embed_tokens, lm_head, embed_vision.embedding_projection) keep their explicit affine overrides.
imatrix AWQ pre-scaling amplifies important weight channels and fuses inverse scales into preceding layer norms (zero inference overhead).
| Parameter | Value |
|---|---|
| Total parameters | ~31B (fully dense — all parameters active per token) |
| Hidden size | 5,376 |
| Layers | 60 (sliding-window attention) |
| Attention heads | 32 (16 KV heads, GQA 2:1) |
| Head dimension | 256 |
| MLP intermediate size | 21,504 |
| Vocab size | 262,144 |
| Max context | 131,072 tokens |
| Vision | yes (Gemma4ForConditionalGeneration) |
import { loadSession } from '@mlx-node/lm';
const session = await loadSession('./Gemma-4-31B-IT-UD-NVFP4_K_XL-mlx');
for await (const event of session.sendStream('Explain the sliding-window attention design in Gemma-4.', {
config: { maxNewTokens: 2048, temperature: 0.6, reasoningEffort: 'low' },
})) {
if (!event.done) process.stdout.write(event.text);
}
mlx convert \
-i gemma-4-31b-it \
-o Gemma-4-31B-IT-UD-NVFP4_K_XL-mlx \
-q --q-mode nvfp4 --q-recipe unsloth \
--imatrix-path imatrix_unsloth.gguf
Gemma Terms of Use (inherited from base model).