Downloads · 30 days
35
3% of all-time downloads
MaliAir/GLM-5.2-MXFP4-MOE-Q8_0-GGUF
GLM-5.2-MXFP4-MOE-Q8_0-GGUF is a machine learning model from MaliAir. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for llama.cpp. The card lists the license as apache-2.0.
This repository contains a quantized and sharded version of the GLM-5.2 base model (originally in BF16), converted to the GGUF format using llama.cpp.
Downloads · 30 days
35
3% of all-time downloads
All-time downloads
1.2K
Public
Repo size
411 GB
Likes
3
Public
Click a slice to open those files.
.gguf411 GB · 100%
From the Hugging Face model README
This repository contains a quantized and sharded version of the GLM-5.2 base model (originally in BF16), converted to the GGUF format using llama.cpp.
To make the model both efficient and practical for deployment, I applied two distinct processing steps:
To achieve the best balance between model size and generation quality, I used a specialized mixed-precision approach with the llama-quantize tool.
This model uses:
mxfp4_moe for the majority of the tensors (especially the MoE expert layers). This is a specialized 4-bit floating-point format designed to efficiently compress Mixture-of-Experts (MoE) models while preserving reasoning capabilities.Q8_0 specifically for the embedding table (token_embd.weight) and the output layer (output.weight). These layers are notoriously sensitive to quantization—keeping them at higher precision (8-bit) significantly reduces degradation in output quality, especially for long generations and complex reasoning tasks.The exact command used to generate the base quantized file was:
./llama-quantize \
--allow-requantize \
--tensor-type token_embd.weight=Q8_0 \
--tensor-type output.weight=Q8_0 \
GLM-5.2-BF16.gguf \
glm-5.2-mxfp4-moe-q8.gguf \
mxfp4_moe
Inference speed was evaluated on the following enterprise-grade hardware configuration:
Under this setup, the quantized model achieves a generation speed of:
≈ 6.8 tokens per second (t/s)
This benchmark was recorded using llama-server with standard inference settings (batch size = 2048, context length = 98304, with layers offloaded to the GPU via -ngl). The combination of the EPYC 9654's massive memory bandwidth (over 400 GB/s thanks to 12-channel DDR5) and the RTX 5090's computational muscle allows the MoE architecture to run smoothly, fully leveraging the mixed-precision quantization. 392GB DDR5 RAM from the MZ73-LM0 and 30GB GDDR7 RAM from the RTX 5090 are used. We are using this quantized and sharded GLM-5.2 base model within our EASA-compliant HPC environment, purpose-built for Aviation Safety and Compliance at masi-ai.com
At 6.8 t/s, the model provides a fluid, interactive experience suitable for real-time chat applications, complex multi-turn dialogues, and light batch processing—delivering near-instantaneous responses without frustrating delays.
./llama-server -m /path/GLM-5.2-MXFP4-MOE-Q8_0.GGUF -ngl 99 -ot "exps=CPU" -t 80 -tb 96 --host xxx.xxx.xxx.xxx --port 8080 --ctx-size 98304 --batch-size 2048 --ubatch-size 2048 --no-mmap --embeddings --pooling mean -fa on
This quantized and sharded GLM-5.2 base model powers our EASA-compliant High-Performance Computing (HPC) cluster, specifically architected for Aviation Safety and Compliance workflows at masi-ai.com.
KHM