GLM-5.2-MXFP6

GLM-5.2-MXFP6

Weight-only MXFP6 (OCP Microscaling FP6, E2M3) quantization of zai-org/GLM-5.2.

Only the routed MoE expert weights are cast to FP6. Everything else (attention, the router, the shared expert, the dense layers, token embeddings, the LM head, the MTP layer, and all activations) is kept in native BF16.

To our knowledge this is the first MXFP6 quantization of GLM-5.2 on the Hub.

• Base model: zai-org/GLM-5.2 (BF16, ~753B total / ~39B active, glm_moe_dsa architecture, 78 layers + 1 MTP layer, 256 routed + 1 shared expert per MoE layer).
• Quantized with: AMD Quark 0.12.post1, --file2file_quantization (round-to-nearest), on a single AMD Instinct MI300X (ROCm 7.2.4).
• Format: OCP MXFP6-E2M3, block size 32, E8M0 shared scales, weight-only (BF16 activations).
• Size: 282 safetensors shards, roughly 623 GB (down from about 1507 GB BF16).

Precision layout

57,600 expert weight tensors are quantized (75 MoE layers x 256 experts x 3 projections), each with a matching E8M0 weight_scale; every other weight passes through unchanged.

Reproduction

Produced with the stock Quark mxfp6_e2m3 scheme (file2file, RTN), then one config edit to make it weight-only:

# AMD Quark 0.12, examples/torch/language_modeling/llm_ptq/
python quantize_quark.py \
    --model_dir zai-org/GLM-5.2 \
    --output_dir GLM-5.2-MXFP6 \
    --quant_scheme mxfp6_e2m3 \
    --exclude_layers "*self_attn*" "*mlp.gate" "*shared_experts*" \
        "*mlp.gate_proj" "*mlp.up_proj" "*mlp.down_proj" \
        "*lm_head*" "*embed*" "*layers.78*" \
    --file2file_quantization \
    --model_export hf_format \
    --skip_evaluation

Then, in the emitted config.json, set quantization_config.global_quant_config.input_tensors to null so the checkpoint declares weight-only quantization (FP6 weights, BF16 activations). The weight bytes are identical either way; file2file never runs a forward pass, so this edit is purely the metadata declaration.

Usage

The checkpoint uses the standard Quark on-disk format (quantization_config.quant_method = "quark", weight-only w_mxfp6_e2m3). FP6 weights are dequantized to BF16 at load on GPUs without native FP6 support. Native FP6 compute requires CDNA4 (MI350/MI355), so on MI300X and other GPUs it runs as dequant-to-BF16, meaning the benefit there is memory footprint rather than throughput.

vLLM (ROCm), adjust --tensor-parallel-size to your GPU count:

export VLLM_ROCM_USE_AITER=1
vllm serve Unrvl22/GLM-5.2-MXFP6 \
    --tensor-parallel-size 8 \
    --quantization quark \
    --trust-remote-code

Serving status: this repository is the checkpoint; end-to-end serving has not been validated on this exact build. Some engines may still lack GlmMoeDsa support, and the MTP layer is emitted as model.layers.78.* (SGLang expects a model.decoder.* prefix, a known name remap). Neither affects the weights in this repo.

Limitations

• Inherits all capabilities, biases, and failure modes of the base zai-org/GLM-5.2. Quantization can introduce small additional output deviations.
• No throughput benefit on pre-CDNA4 hardware; the benefit there is reduced memory footprint only.

License and attribution

Released under the MIT License, inheriting the license of the base model zai-org/GLM-5.2. This is a derivative work (weight quantization only); all credit for the underlying model belongs to Z.ai / Zhipu AI.

References

• Base model: https://huggingface.co/zai-org/GLM-5.2
• AMD Quark: https://quark.docs.amd.com/
• AMD MXFP4 reference quant: https://huggingface.co/amd/GLM-5.2-MXFP4
• OCP Microscaling (MX) formats specification: https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf