MiMo-V2.6-Distill-Qwen-9B-GGUF

MiMo-V2.6-Distill-Qwen-9B GGUF

Locally validated GGUF quants of XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B.

Files

Source revision: f2773fb482ac3dd047a4af4003b86e56b7225d0d

llama.cpp conversion/test commit: ce8caa6

IQ4_XS SHA256: bc2d7e526df4bcc24fd5b1554c1f7c7062569706e2ce7835a8ebbfb6087e00e4

Q4_K_M SHA256: 7ecfcb18dc822ca46da9acf138af7db995a9b9db3a0c4d19c6c438b1dde72a08

iMatrix

The IQ4_XS quant uses a task-mixed importance matrix instead of a Wiki-only calibration.

Calibration mix includes general text, real llama.cpp C/C++ and Python, shell/config/JSON, OWASP security documentation, and MiMo-style chat/tool-call examples.

• context per chunk: 512 tokens
• chunks processed: 64
• importance-matrix entries: 248
• layers covered: 32/32
• minimum tensor active rate: 97.78%
• median tensor active rate: 100%
• median normalized activation entropy: 81.82%
• calibration text SHA256: f4ce34541473d074e9e355772dee137dafe59330c6edb2c94f2397a19babfa11
• iMatrix SHA256: a9163b33e811a68f077a916d6b1c067e06de3590f075808dd60ce950cdaece66

The calibration source text is not redistributed here. The derived iMatrix is included for additional quantization experiments.

The final IQ4_XS mixture keeps output.weight at Q6_K and automatically promotes selected attention tensors to Q5_K while quantizing the main body to IQ4_XS.

Conversion note

The upstream config declares mtp_num_hidden_layers: 1, but the released safetensors contain no MTP/NextN tensors. A normal conversion produced metadata that made llama.cpp request a nonexistent blk.32.attn_norm.weight.

Both published quants therefore use a BF16 GGUF converted with --no-nextn. The resulting model reports 32 blocks and contains no qwen35.nextn_predict_layers metadata.

Chat template

The original MiMo v2.6 Jinja chat template is embedded byte-for-byte.

Template SHA256: 59a64ebb4df6d1489d09a91267cf3ceb106162d4a893c4f84833cfb8c897ff63

Do not override it with generic ChatML. Use llama.cpp with Jinja enabled.

The upstream quickstart explicitly enables thinking. Local testing found that some prompts can expose malformed reasoning boundaries such as a literal </think>. The same behavior was reproduced with the BF16 baseline, so it is not specific to the IQ4_XS quant. enable_thinking=false was also unreliable in local testing.

Recommended upstream sampling: temperature 0.6, top-k 20, top-p 0.95.

Local validation

The final IQ4_XS file was tested on an RTX 4090 with llama.cpp before upload:

• model load: PASS
• embedded template vs upstream template: exact match
• arithmetic: PASS
• code generation: PASS
• short cybersecurity explanation: PASS
• general-knowledge explanation: PASS
• multi-turn history: PASS
• structured function/tool call: PASS
• bounded smoke tests: no repetition loop observed

llama.cpp

llama-server -m MiMo-V2.6-Distill-Qwen-9B-IQ4_XS.gguf -c 8192 -ngl 999 --jinja

For OpenAI-compatible chat requests, keep the embedded template and pass chat_template_kwargs: {"enable_thinking": true}.

Upstream

Model: https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B

At the time these quants were prepared, the upstream Distill-Qwen-9B repository did not declare a license in its model metadata. Check the upstream repository for current usage terms.