GLM-5.2-Quark-MXFP4-AttnFP8

Model Overview

• Model Architecture: GLM-5.2
  • Input: Text
  • Output: Text
• Supported Hardware Microarchitecture: AMD MI350/MI355
• ROCm: 7.0.0
• PyTorch: 2.9.0
• Transformers: 5.13.1
• Operating System(s): Linux
• Inference Engine: vLLM
• Model Optimizer: AMD-Quark (V0.12)
  • Weight quantization: OCP MXFP4, Static; self_attn Per-channel, FP8E4M3, Static
  • Activation quantization: OCP MXFP4, Dynamic; self_attn Per-token, FP8E4M3, Dynamic

This model was built with GLM-5.2 model by applying AMD-Quark for MXFP4 and PTPC FP8 quantization.

Model Quantization

The model was quantized from zai-org/GLM-5.2 using AMD-Quark.

Quantization scripts:

cd Quark/examples/torch/language_modeling/llm_ptq/
python quantize_quark.py \
  --model_dir zai-org/GLM-5.2 \
  --output_dir "$OUTPUT_DIR" \
  --quant_scheme mxfp4 \
  --layer_quant_scheme "*self_attn*" ptpc_fp8 \
  --exclude_layers \
      "*lm_head" \
      "*embed_tokens" \
      "*eh_proj" \
      "*mlp.gate" \
      "*self_attn.indexer.weights_proj" \
  --file2file_quantization

Deployment

Use with vLLM

This model can be deployed efficiently using vLLM.

Evaluation

The model was evaluated on GSM8K and GPQA-Diamond-CoT benchmarks.

Accuracy

<table> <tr> <td><strong>Benchmark</strong> </td> <td><strong>zai-org/GLM-5.2</strong> </td> <td><strong>amd/GLM-5.2-Quark-MXFP4-AttnFP8 (this model)</strong> </td> <td><strong>Recovery</strong> </td> </tr> <tr> <td>GSM8K (flexible-extract) </td> <td>96.46 </td> <td>95.80 </td> <td>99.3% </td> </tr> <tr> <td>GPQA-Diamond-CoT (flexible-extract) </td> <td>69.53 </td> <td>68.35 </td> <td>98.3% </td> </tr> </table>

Reproduction

The above evaluation results were obtained using the lm-evaluation-harness framework, based on the Docker image rocm/vllm-dev:nightly_main_20260712, with vLLM and lm-eval compiled and installed from source.

export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_FP8BMM=0
export VLLM_ROCM_USE_AITER_FP4BMM=0

lm_eval \
        --model vllm \
        --model_args "pretrained=amd/GLM-5.2-Quark-MXFP4-AttnFP8,tensor_parallel_size=4,dtype=auto,quantization=quark,gpu_memory_utilization=0.9,max_model_len=40960,trust_remote_code=True" \
        --tasks gsm8k gpqa_diamond_cot_zeroshot \
        --apply_chat_template \
        --gen_kwargs "max_gen_toks=32768,temperature=1.0,top_p=0.95,do_sample=True" \
        --batch_size auto

License

Modifications Copyright(c) 2026 Advanced Micro Devices, Inc. All rights reserved.