MiMo-V2.6-Distill-Qwen-9B-GGUF

MiMo-V2.6-Distill-Qwen-9B · GGUF Q8_0 and calibrated Q6_K

GGUF conversions of XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B, Xiaomi MiMo's agentic SFT checkpoint based on Qwen3.5-9B. This repository provides standard Q8_0, imatrix-calibrated Q6_K, and a shared F16 vision projector for llama.cpp.

Status: converted, integrity-checked, and tested on Apple M4 Max. Both quantizations passed a text-generation sanity check and a small matched held-out perplexity comparison. Q6_K with the projector also passed an image-input sanity check.

At a glance

Q8_0 does not use imatrix weights. At the pinned llama.cpp revision, quantize_q8_0 explicitly ignores importance weights. Q6_K uses the weighted quantization path. The token embeddings and output matrix use ordinary Q6_K quantization without collected importance weights; the remaining 248 quantized tensors use the imatrix.

Files

Sizes are decimal GB and describe files on disk, not total runtime memory.

The original BF16 reference and Hugging Face weight snapshot were used for conversion and validation; they are not duplicated in this repository. The imatrix is for quantization and is not needed for inference.

Usage

Use a recent llama.cpp build. The exact tested revision is listed below. Download the desired quantization with the Hugging Face CLI; access to this private repository requires an authenticated account with permission.

hf download RESMP-DEV/MiMo-V2.6-Distill-Qwen-9B-GGUF \
  MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf \
  --local-dir ./MiMo-V2.6-Distill-Qwen-9B-GGUF

llama-cli \
  -m ./MiMo-V2.6-Distill-Qwen-9B-GGUF/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf \
  -ngl 99 -c 8192

For Q8_0, substitute its filename in both commands. The -c 8192 setting is a usage example; validation used the shorter contexts described below.

Image input

hf download RESMP-DEV/MiMo-V2.6-Distill-Qwen-9B-GGUF \
  mmproj-MiMo-V2.6-Distill-Qwen-9B-F16.gguf \
  --local-dir ./MiMo-V2.6-Distill-Qwen-9B-GGUF

llama-cli \
  -m ./MiMo-V2.6-Distill-Qwen-9B-GGUF/MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf \
  --mmproj ./MiMo-V2.6-Distill-Qwen-9B-GGUF/mmproj-MiMo-V2.6-Distill-Qwen-9B-F16.gguf \
  --image ./image.png \
  -ngl 99 -c 2048 --single-turn \
  --chat-template-kwargs '{"enable_thinking":false}' \
  -p "Describe this image."

The original template supports the enable_thinking setting. The generation sanity checks explicitly disabled thinking. Tool-call parsing and serving behavior have not been evaluated in this build.

Calibration and conversion

Calibration used eaddario/imatrix-calibration, file combined_all_micro.parquet, containing packed code, tool-use, math, and multilingual text. The content column was exported to UTF-8 text with a final newline. Of 239,567 model-tokenizer tokens, the first 204,800 were used for calibration.

llama-imatrix ran on the BF16 reference with full Metal offload, context 2,048, batch 2,048, microbatch 512, eight CPU threads, and --no-ppl. The 248 weight entries are stored as 496 statistics tensors: a sum-of-squares array and an observation count per entry. All entries have positive counts and finite statistics. The projector is a separate F16 conversion and was not calibrated with images.

The published source config advertises one MTP layer, but its tensor index contains no MTP weights. Conversion used upstream's --no-mtp option to produce the correct 32-layer GGUF without NextN metadata. No llama.cpp source patch was required.

# After converting the pinned source to model-BF16.gguf with --no-mtp:
llama-imatrix -m model-BF16.gguf -f combined_all_micro.txt \
  -o imatrix.gguf -ngl 99 -c 2048 -b 2048 -ub 512 \
  --chunks 100 --no-ppl --output-frequency 10 -t 8

llama-quantize --imatrix imatrix.gguf \
  model-BF16.gguf MiMo-V2.6-Distill-Qwen-9B-Q6_K.gguf Q6_K 8

llama-quantize \
  model-BF16.gguf MiMo-V2.6-Distill-Qwen-9B-Q8_0.gguf Q8_0 6

Validation

Matched checks on Apple M4 Max used the same held-out text and runtime for BF16, Q8_0, and Q6_K.

The held-out text begins at token 212,992, separated from calibration by an 8,192-token gap. Perplexity used eight 512-token chunks, scoring the second half of each chunk: 2,048 scored tokens total. Reported uncertainty is included in provenance.json.

The arithmetic prompt was What is 15% of 240? Answer with only the number., using greedy generation, the original template, thinking disabled, and a 2,048-token context. Q6_K plus the F16 projector also correctly answered Red for a solid-red test image through standard llama-cli.

These are small sanity checks. The lower Q6_K perplexity on this sample does not establish a general quality improvement. No agentic, coding, tool-use, long-context, or comprehensive vision benchmark has been run on these quantizations. Upstream model-card benchmark scores have not been remeasured here.

Reproducibility and integrity

• Source model revision: f2773fb482ac3dd047a4af4003b86e56b7225d0d; all 17 source files checksum-verified.
• llama.cpp revision: ff0dbb975e93a9a2899efa34bdd32d1c5cfbc183, freshly cloned and built on 2026-09-21.
• Calibration dataset revision: e87ed55dcba9d9c3a3e41539f3e728e981b1daa4.
• Calibration text SHA-256: fdb2d41abf04a2fb207502741a561a5a9ab385eb0c44a450eae676c410955946.
• Imatrix SHA-256: 47ef9d566ce88359dce7677e3b1b7c9f1f045739d8a164929280c2533bfe63e1.
• Both quantizations retain the BF16 reference's 427 tensor names/shapes and original embedded chat template.

See SHA256SUMS for artifact hashes, provenance.json for structured receipts, and chat_template.jinja for the source template.

Credits

Original model and distillation: Xiaomi MiMo. Base architecture: Qwen3.5. GGUF conversion and quantization runtime: ggml-org/llama.cpp. Calibration corpus: eaddario/imatrix-calibration. Quantized builds and local verification: RESMP-DEV.

The pinned source model card does not specify a license; no license from another RESMP-DEV model has been assigned to this derivative. Consult the upstream repository for its terms.