Downloads · 30 days
42
13% of all-time downloads
dpcpw/Mellum2-12B-A2.5B-Thinking-GGUF-mixed-q4-q8
Mellum2-12B-A2.5B-Thinking-GGUF-mixed-q4-q8 is a text generation model from dpcpw. Use it when you need the model to write or continue text. It is set up for gguf. The card lists the license as apache-2.0.
This is a mixed GGUF quantization of JetBrains/Mellum2-12B-A2.5B-Thinking, inspired by jedisct1/Mellum2-12B-A2.5B-Thinking-mlx-4bit.
Downloads · 30 days
42
13% of all-time downloads
All-time downloads
334
Public
Repo size
17.1 GB
Likes
2
Public
Click a slice to open those files.
.gguf8.5 GB · 100%
From the Hugging Face model README
This is a mixed GGUF quantization of JetBrains/Mellum2-12B-A2.5B-Thinking, inspired by jedisct1/Mellum2-12B-A2.5B-Thinking-mlx-4bit.
It was quantized from the BF16 GGUF in JSchneemann/Mellum2-12B-A2.5B-Thinking-GGUF.
The goal is to keep formatting-sensitive tensors at higher precision while quantizing the large MoE expert tensors aggressively:
q8_0q8_0attn_q, attn_k, attn_v, attn_output): q8_0ffn_gate_inp): unchanged f32 in the source GGUFQ4_K_M, with llama.cpp fallbacks where tensor shapes do not support the requested K-quant typeCommand used:
llama-quantize \
--token-embedding-type q8_0 \
--output-tensor-type q8_0 \
--tensor-type attn_q=q8_0 \
--tensor-type attn_k=q8_0 \
--tensor-type attn_v=q8_0 \
--tensor-type attn_output=q8_0 \
--tensor-type ffn_gate_inp=q8_0 \
Mellum2-12B-A2.5B-Thinking.BF16.gguf \
Mellum2-12B-A2.5B-Thinking.moe-q4-attn-q8.gguf \
Q4_K_M
Note: this is not bit-identical to the MLX affine 4-bit quantization. It applies the same high-level mixed-precision idea using llama.cpp/GGUF quantization types.
Community-reported llama-perplexity results on WikiText-2 with context size 512 show this mixed quantization preserving substantially more quality than standard 4-bit Mellum2 GGUF quants at a similar size. These numbers were reported by a third party and have not yet been independently re-run by the uploader.
| Quantization | Perplexity | Size | Notes |
|---|---|---|---|
| BF16 original baseline | 9.4037 +/- 0.06784 | ~25.0 GB | Reference baseline |
| Q8_0 | 9.3657 +/- 0.06734 | 12.9 GB | Highest-quality standard quant |
| Q5_K_M | 9.4490 +/- 0.06807 | 9.21 GB | Standard K-quant |
| Mixed Q4 experts + Q8 attention/embeddings/output | 9.5392 +/- 0.06916 | ~7.9 GB | This repository |
| Q6_K | 9.7329 +/- 0.07207 | 10.9 GB | Worse than expected on this model in the reported run |
| Q4_K_M | 9.7410 +/- 0.07128 | 8.07 GB | Standard K-quant |
| Q4_K_S | 9.8269 +/- 0.07248 | 7.4 GB | Standard K-quant |
| Q4_0 | 18.6689 +/- 0.17737 | 16.91 GB | Reported as unsuitable for this MoE model |
The result suggests that keeping attention, embeddings, and output tensors at Q8 while quantizing the large MoE expert tensors to Q4_K_M is a strong size/quality trade-off for this model, especially for VRAM-limited systems.
Mellum2 requires llama.cpp support for the mellum architecture. At upload time this support is from ggml-org/llama.cpp PR #23966, or a newer llama.cpp release if it has been merged.
Recommended sampling settings from JetBrains:
temperature=0.6
top_p=0.95
top_k=20
Apache 2.0, inherited from the original JetBrains model.