Kimi-K3-Quark-MXFP4-AttnFP8

Model Overview

• Model Architecture: KimiK3ForConditionalGeneration
  • Input: Text, Image, Video
  • Output: Text
• Supported Hardware Microarchitecture: AMD MI350/MI355
• ROCm: 10.0.0
• PyTorch: 2.11.0
• Transformers: 5.19.0
• Operating System(s): Linux
• Inference Engine: SGLang
• Model Optimizer: AMD-Quark
  • experts: OCP MXFP4
  • self_attn: PTPC-FP8
  • shared_experts: BF16

Model Quantization

The model was quantized from moonshotai/Kimi-K3 using AMD-Quark. Expert weights and activations are quantized to OCP MXFP4 and self-attention to PTPC-FP8, while the shared experts are excluded from quantization and stay in BF16.

Quantization scripts:

from quark.torch import LLMTemplate, ModelQuantizer
# --- Register template ---
kimi_k3_template = LLMTemplate(
    model_type="kimi_k3",
    kv_layers_name=[
        "*self_attn.k_proj",
        "*self_attn.v_proj",
        "*self_attn.kv_a_proj_with_mqa",
    ],
    q_layer_name=[
        "*self_attn.q_proj",
        "*self_attn.q_a_proj",
        "*self_attn.q_b_proj",
    ],
    exclude_layers_name=[
        "*block_sparse_moe.gate*",
        "*block_sparse_moe.routed_expert_down_proj*",
        "*block_sparse_moe.routed_expert_up_proj*",
        "*block_sparse_moe.shared_experts*",
        "*lm_head*",
        "*vision_tower*",
        "*mm_projector*",
        "*self_attention_res_proj*",
        "*mlp_res_proj*",
        "*output_attn_res_proj*",
        "*self_attn.*_conv1d*",
    ],
)
LLMTemplate.register_template(kimi_k3_template)
print(f"[INFO]: Registered template '{kimi_k3_template.model_type}'")
# --- Configuration ---
model_dir = "moonshotai/Kimi-K3"
output_dir = "amd/Kimi-K3-Quark-MXFP4-AttnFP8"
quant_scheme = "mxfp4"
#   --layer_quant_scheme "*self_attn*" ptpc_fp8
layer_config = {
    "*self_attn*": "ptpc_fp8",
}
# --- Build quant config from template ---
template = LLMTemplate.get("kimi_k3")
quant_config = template.get_config(
    scheme=quant_scheme,
    layer_config=layer_config,
)
# --- File-to-file quantization ---
quantizer = ModelQuantizer(quant_config)
quantizer.direct_quantize_checkpoint(
    pretrained_model_path=model_dir,
    save_path=output_dir,
)
print(f"[INFO]: Quantization complete. Output saved to {output_dir}")

Evaluation

The model is evaluated using the SGLang framework.

Accuracy

Reproduction

The GSM8K results were obtained using lm_eval and the SGLang framework, based on the Docker image rocm/sgl-dev:v0.5.22-rocm10-mi35x-20261019, with SGLang pre-installed inside the container, and lm_eval installed from source.

Launching server

SGLANG_USE_AITER=1 \
SGLANG_AITER_K3_OPT=1 \
AITER_SITUV2_A4W4=1 \
python -m sglang.launch_server \
  --model-path amd/Kimi-K3-Quark-MXFP4-AttnFP8 \
  --trust-remote-code \
  --tp-size 8 \
  --dcp-size 8 \
  --dcp-comm-backend a2a \
  --prefill-attention-backend aiter \
  --decode-attention-backend aiter \
  --kv-cache-dtype fp8_e4m3 \
  --dtype bfloat16 \
  --mem-fraction-static 0.90 \
  --cuda-graph-max-bs-decode 256 \
  --reasoning-parser kimi_k3 \
  --tool-call-parser kimi_k3 \
  --model-loader-extra-config '{"enable_multithread_load": true}' \
  --host 127.0.0.1 \
  --port 30000

Evaluating model in a new terminal

lm_eval --model local-completions \
  --model_args "model=amd/Kimi-K3-Quark-MXFP4-AttnFP8,base_url=http://127.0.0.1:30000/v1/completions,num_concurrent=64,max_retries=3,tokenized_requests=False,tokenizer_backend=None" \
  --tasks gsm8k \
  --num_fewshot 5 \
  --batch_size 1

License

Modifications Copyright(c) 2026 Advanced Micro Devices, Inc. All rights reserved.