Kimi-K2.6-MXFP4

Kimi-K2.6-MXFP4

MXFP4 (OCP Microscaling FP4, group-size 32 with E8M0 scales) quantization of moonshotai/Kimi-K2.6, produced with AMD Quark 0.11.

The recipe is transferred verbatim from AMD's reference amd/Kimi-K2.5-MXFP4 (same DeepseekV3 backbone, same MLA dimensions, same 384+1 MoE topology as K2.6), with no calibration and no hand-tuning beyond the published exclude list.

Coverage

Quantized to MXFP4:

• All 384 routed experts across all 60 MoE layers (mlp.experts.*.{gate,up,down}_proj)
• MoE routers (mlp.gate.weight)

Kept at BF16 (excluded):

• lm_head
• All MLA projections (self_attn.*: q_a, q_b, kv_a, kv_b, o)
• Shared experts (shared_experts.*)
• Dense MLP on layer 0 (mlp.{gate,up,gate_up,down}_proj — does not match routed-expert patterns)
• Multimodal vision tower and projector

This matches the exclude list AMD used for K2.5-MXFP4.

Quality

GSM8K 3-shot (n=1319), lm_eval via /v1/completions, 3-seed mean ± stdev (seeds 42 / 1234 / 7777):

Δ = +0.40 pp (FP8 higher), z = 1.26 — not statistically significant at 95%. Matches the 0–1 pp MXFP4-vs-reference band reported for AMD's K2.5-MXFP4 and other published MXFP4 quants on reasoning benchmarks.

The "upstream reference" for this comparison was produced offline by dequantizing upstream K2.6's INT4 pack-quantized experts to BF16 (bit-exact against an independently-produced BF16 staging checkpoint), then quantizing to per-channel FP8 E4M3 for serving on MI350X. The non-expert layers were passed through as BF16 unchanged.

Architecture

• Backbone: DeepseekV3-family (wrapped as KimiK25ForConditionalGeneration)
• 61 layers (1 dense MLP + 60 MoE)
• 384 routed + 1 shared expert, top-8, moe_intermediate_size=2048
• MLA: kv_lora_rank=512, q_lora_rank=1536, qk_nope=128, qk_rope=64, v_head=128, 64 heads
• RoPE: YARN, factor 64, original 4k, max 262144
• Vision tower: MoonViT (can be bypassed for text-only serving)

Serving

Tested and working on:

• ATOM (AMD MI350X / gfx950) — loads via Mxfp4MoEMethod in atom/model_ops/moe.py. Start with:

  python -m atom.entrypoints.openai_server \
      --model wafer-ai/Kimi-K2.6-MXFP4 \
      --trust-remote-code -tp 8 \
      --kv_cache_dtype fp8 \
      --gpu-memory-utilization 0.90 \
      --max-model-len 16384
  

• vLLM built for ROCm — loads via the quark quant loader path (built into upstream vLLM). We tested inside AMD's rocm/atom-dev:vllm-latest container. --trust-remote-code required.

• SGLang (ROCm build) — loads cleanly; confirm your SGLang build supports the quark/compressed-tensors MXFP4 path on ROCm.

Memory footprint at TP=8: ~70 GB/GPU weights + overhead; fits comfortably on 8× MI350X VF (288 GB HBM each).

Limitations

• Quark 0.11 is pinned: 0.12+ changes pack_method defaults (→ reorder) and may produce a checkpoint that does not load under the current AMD serving stacks. Reproduce with amd-quark==0.11.*.
• Not tested on MI355X or H100/B200 — recipe is MI350X-validated.
• Vision serving path has not been end-to-end validated in this repo; text-only workloads are the primary target.
• Performance numbers (tok/s) are stack-specific; see your serving documentation for details.

Reproducing

Recipe (config.json quantization_config block):

{
  "quant_method": "quark",
  "version": "0.11+4a34634b4a",
  "global_quant_config": {
    "weight":        {"dtype": "fp4", "qscheme": "per_group", "ch_axis": -1, "group_size": 32, "scale_format": "e8m0", "scale_calculation_mode": "even", "observer_cls": "PerBlockMXObserver", "is_dynamic": false},
    "input_tensors": {"dtype": "fp4", "qscheme": "per_group", "ch_axis": -1, "group_size": 32, "scale_format": "e8m0", "scale_calculation_mode": "even", "observer_cls": "PerBlockMXObserver", "is_dynamic": true}
  },
  "exclude": [
    "lm_head",
    "re:.*self_attn.*",
    "re:.*shared_experts.*",
    "re:.*mlp\\.(gate|up|gate_up|down)_proj.*",
    "re:mm_projector.*",
    "re:vision_tower.*"
  ],
  "algo_config": null,
  "export": {"pack_method": "reorder", "weight_format": "real_quantized"}
}

Attribution and License

Weights are derived from moonshotai/Kimi-K2.6, licensed under the Modified MIT (Kimi) license. The LICENSE file is preserved in this repository verbatim from the upstream release; all upstream attributions and notices apply.

For upstream model details, training data, capabilities, and intended use, see the original model card at moonshotai/Kimi-K2.6.

Citation

If this quantization is useful in your work, please cite upstream Kimi-K2.6 and, optionally, this quant:

@misc{moonshot_kimi_k26,
  title = {Kimi K2.6},
  author = {Moonshot AI},
  howpublished = {\url{https://huggingface.co/moonshotai/Kimi-K2.6}},
  year = {2026}
}
@misc{waferai_kimi_k26_mxfp4,
  title = {Kimi-K2.6-MXFP4},
  author = {Wafer AI},
  howpublished = {\url{https://huggingface.co/wafer-ai/Kimi-K2.6-MXFP4}},
  year = {2026}
}