Downloads · 30 days
79
19% of all-time downloads
MiMo-V2.6-Distill-Qwen-9B-GGUF
jmarceno/MiMo-V2.6-Distill-Qwen-9B-GGUF
MiMo-V2.6-Distill-Qwen-9B-GGUF is a machine learning model from jmarceno. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
Two dynamic quantizations of XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B (MIT), plus the importance matrix and the exact recipes.
All-time downloads
422
Public
Repo size
9.7 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.gguf9.7 GB · 100%
At a glance
- License
- mit
- Access
- Public
- Created
- Sep 22, 2026
- Updated
- Sep 22, 2026
- SHA
- 660841a9
Base models
Model card
From the Hugging Face model README
MiMo-V2.6-Distill-Qwen-9B — dynamic GGUFs (+ the imatrix)
Two dynamic quantizations of XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B (MIT), plus the importance matrix and the exact recipes.
The model is a dense Qwen3.5-9B SFT distill (GDN linear attention + full attention every 4th layer, 32 layers, 262144 context). The source config advertises one MTP layer, but the released safetensors contain no mtp.* weights, so these GGUFs were converted with --no-mtp (block count 32).
Files
| file | size | bpw | what |
|---|---|---|---|
MiMo-V2.6-Distill-Qwen-9B-DYN-DIQ4XS-GU2XXS-Q8_0.gguf | 5.42 GiB | 5.20 | Xing4-literal hybrid: attention/SSM stay Q8_0, down@IQ4_XS, gate/up@IQ2_XXS |
MiMo-V2.6-Distill-Qwen-9B-DYN-A4-DIQ3XXS-GU2XXS-Q5_K.gguf | 3.63 GiB | 3.48 | tighter mix aimed at ~3.51 BPW: attention/SSM Q4_K, down@IQ3_XXS, gate/up@IQ2_XXS |
imatrix-MiMo-V2.6-Distill-Qwen-9B-820chunks.gguf | 5.0 MiB | — | imatrix, 820 chunks × 512 tokens |
sha256:
83d0af7279f61ff2e5d0847802f4019dd959b53ec2475f94553f4c0fbad10019 MiMo-V2.6-Distill-Qwen-9B-DYN-DIQ4XS-GU2XXS-Q8_0.gguf
4fabc7ac103384dbed0221817a2dde3a29a7165830febe3b61d98bed317653ca MiMo-V2.6-Distill-Qwen-9B-DYN-A4-DIQ3XXS-GU2XXS-Q5_K.gguf
565d2f416187f2f4915220600fe50ed40d80067540fc0b0286a37cca527700e5 imatrix-MiMo-V2.6-Distill-Qwen-9B-820chunks.gguf
The down projection gets more bits than gate/up because its output feeds the residual stream unattenuated — the same family split used for Xing4.0-29B-A4B. On this dense model, leaving attention at Q8_0 lands at 5.20 BPW, so the second recipe lowers attention/SSM to reach the ~3.5 BPW size.
Measured
Held-out WikiText-2 raw test, CPU llama.cpp, -c 512 --chunks 16 --threads 8 (English-only corpus; relative A/B, not an absolute quality score):
| artifact | bpw | perplexity |
|---|---|---|
| 5.20 hybrid | 5.20 | 12.2080 ± 0.532 |
| 3.48 mix | 3.48 | 12.4082 ± 0.541 |
The smaller file is about 1.6% higher perplexity on this short English probe.
Running it
Needs a llama.cpp build with Qwen3.5 (qwen35) support.
llama-server \
-m MiMo-V2.6-Distill-Qwen-9B-DYN-A4-DIQ3XXS-GU2XXS-Q5_K.gguf \
--alias mimo-v2.6-distill-qwen-9b -c 131072 --parallel 1 \
--n-gpu-layers 99 -ctk q4_0 -ctv q4_0 -fa on \
--jinja --temp 1.0 --top-p 0.95
Text-only GGUF. The vision tower was not converted.
Reproducing
See quantize-recipe.txt, quantize-recipe-5.20bpw.txt, and quantize-recipe-3.51bpw.txt. Outline:
convert_hf_to_gguf.py --outtype bf16 --no-mtp- plain
IQ4_NLworking model (fits a single 12 GB GPU; used only to collect activations) llama-imatrix -c 512 --chunks 820over a mixed EN/ZH/code calibration corpusllama-quantize --imatrix ...with the recipe file
Quantized 2026-09-22. Base model: MiMo-V2.6-Distill-Qwen-9B, MIT, Xiaomi MiMo.
- License
- mit
- Languages
- en, zh
- Created
- Sep 22, 2026
- Updated
- Sep 22, 2026