GLM-5.2-Int4-Int8Mix-FP8KV

GLM-5.2-Int4-Int8Mix-FP8KV

Base model: zai-org/GLM-5.2 then QuantTrio/GLM-5.2-Int4-Int8Mix

This repo adds MLA FP8 KV scales to QuantTrio/GLM-5.2-Int4-Int8Mix using a VLLM custom tool and calibration dataset provided in agentvue/vllm-plus repo.

So that older NVIDIA hardware (like Ampere cards / SM8x or even Blackwell SM12x) can run this repo using vllm-plus with TRITON_MLA_SPARSE backend and FP8 KV CACHE.

【Quantization Policy】

Reference quantized weights from QuantTrio/GLM-5.2-Int4-Int8Mix have not been modified, so please see QuantTrio - Quantization Policy for more details about LLM quant recipe and accuracy.

【Dependencies / Installation】

As of 2026-07-17, this model has been verified on a mixed GPUs setup (SM8x/SM12x TP4 PP9) with a Python 3.12 virtual environment, agentvue/vllm-plus 0.24.0, and Transformers 5.12.1.

# CUDA Toolkit 13.3 and related NVIDIA drivers (assuming all previous cuda toolkit and drivers have been already cleaned up)

wget https://developer.download.nvidia.com/compute/cuda/13.3.0/local_installers/cuda_13.3.0_610.43.02_linux.run
sudo sh cuda_13.3.0_610.43.02_linux.run

# PYTHON 3.12

pyenv install 3.12.11
pyenv virtualenv 3.12.11 v312
pyenv activate v312

# GLM-5.2-Int4-Int8Mix-FP8KV

pip install huggingface-hub
mkdir -p $HOME/llm/models/GLM-5.2-Int4-Int8Mix-FP8KV
cd $HOME/llm/models
hf download agentvue/GLM-5.2-Int4-Int8Mix-FP8KV --local-dir GLM-5.2-Int4-Int8Mix-FP8KV

# OTHER DEPENDENCIES AND CONFIG (RAY, TRANSFORMERS, SWAPOFF)

pip install ray==2.56.0 # tested with 2.56.0 but newer versions can work too
pip install transformers==5.12.1 # tested with 5.12.1 but newer versions can work too 

sudo swapoff -a && free -h # OPTIONAL: Disable swap space to avoid vllm using this space (even if there's enough RAM left), as it might slow down vllm weights loading

rm -r .cache/vllm && rm -r /tmp/torchinductor*   # OPTIONAL: to clean previous/older tmp vllm files and can avoid some vllm compilation loading errors

# AGENTVUE/VLLM-PLUS V0.24.0

cd $HOME/llm
git clone --branch plus/v0.24.0 --single-branch https://github.com/agentvue/vllm-plus.git

cd vllm-plus
pip install -r requirements/build/cuda.txt
pip install -r requirements/cuda.txt

MAX_JOBS=48 pip wheel --no-build-isolation -v -w dist . 2>&1 | tee build.log # replace MAX_JOBS=48 by higher values or remove this tag if your build setup has enough ram for each CPU thread
pip install ./dist/vllm*.whl

# NCCL 2.30.7 and its PATCH (OPTIONAL, for mixed GPU architecture SM86/SM120 and hybrid interconnection between machines: IB-ethernet)

cd
mkdir nccl-matrix
python -m pip install --target ~/nccl-matrix/nccl-cu13-2.30.7 nvidia-nccl-cu13==2.30.7 # default 2.28.9 coming from cuda toolkit v13.3 has some bugs and crash with vllm using TP and PP on some setups

bash ~/llm/vllm-plus/tools/ib_triangle/build_nccl_peer_rail_mixed.sh

# Copy to other hosts the following built files:
# ~/nccl-2.30.7-peerrail-topo-sm86-sm120.tar.gz
# ~/nccl-2.30.7-peerrail-topo-sm86-sm120.tar.gz.sha256
# Then, install the nccl patch on each other host:
cd "$HOME"
sha256sum -c nccl-2.30.7-peerrail-topo-sm86-sm120.tar.gz.sha256
mkdir -p "$HOME/nccl-matrix/nccl-cu13-2.30.7-peerrail-topo-sm86-sm120"
tar -xzf nccl-2.30.7-peerrail-topo-sm86-sm120.tar.gz \
  -C "$HOME/nccl-matrix/nccl-cu13-2.30.7-peerrail-topo-sm86-sm120"

【vLLM-Plus Startup Command Example】

<i>Note: The following example vllm command has been launch on 3 hosts setup (having each 12 NVIDIA SM8x/SM12x GPUs) using ray. So the ray instances have to be launched before the vllm command and well setting up according to your setups. Same goes for the vllm command below, it should be adapted to your setup.</i>

VLLM_NCCL_SO_PATH=$HOME/nccl-matrix/nccl-cu13-2.30.7/nvidia/nccl/lib/libnccl.so.2 \
LD_LIBRARY_PATH=$HOME/nccl-matrix/nccl-cu13-2.30.7/nvidia/nccl/lib:$LD_LIBRARY_PATH \
LD_PRELOAD=$HOME/nccl-matrix/nccl-cu13-2.30.7/nvidia/nccl/lib/libnccl.so.2 \
VLLM_DISABLE_PYNCCL=0 \
RAY_ADDRESS="$HEAD_ADDRESS" \
VLLM_RAY_WORKER_START_TIMEOUT_S=120 \
VLLM_RAY_WORKER_INIT_TIMEOUT_S=1800 \
VLLM_RAY_ORDERED_NODE_IPS="$ORDERED_NODE_IPS" \
VLLM_RAY_NODE_ENV_VARS_JSON="$NODE_ENV_JSON" \
VLLM_USE_DEEP_GEMM=0 \
VLLM_SPARSE_INDEXER_MAX_LOGITS_MB=128 \
VLLM_LOGGING_LEVEL=DEBUG \
vllm serve "$model_path" \
      --host 0.0.0.0 \
      --port 8000 \
      --served-model-name GLM-5.2 \
      --dtype bfloat16 \
      --quantization compressed-tensors \
      --kv-cache-dtype fp8 \
      --tensor-parallel-size 4 \
      --pipeline-parallel-size 9 \
      --max-model-len 250112 \
      --max-num-batched-tokens 2048 \
      --gpu-memory-utilization 0.956 \
      --no-enable-flashinfer-autotune \
      --max-num-seqs 4 \
      --enable-auto-tool-choice \
      --tool-call-parser glm47 \
      --reasoning-parser glm45 \
      --disable-uvicorn-access-log \
      --attention-backend TRITON_MLA_SPARSE \
      --distributed-executor-backend ray \
      --async-scheduling \
      --chat-template-content-format string \
      --block-size 128 \
      --distributed-timeout-seconds 120 \
      --cpu-distributed-timeout-seconds 120 \
      --safetensors-load-strategy lazy 2>&1 | tee vllm.log

To optimize this command according to your setups, you can check vLLM Official GLM-5.2 Guide.

The first launch may spend several minutes on vLLM compilation and warmup.

【PERF Example】

<i>Note: The following benchmark has been run on the same test lab hardware: 3 hosts setup (having each 12 NVIDIA SM8x/SM12x GPUs and hybrid interconnection using vllm-plus nccl/ray patch: 2 machines interconnected with IB 56G cards and with 10G ethernet to the 3rd machine).</i>

VLLM_LOGGING_LEVEL=DEBUG vllm bench serve \
  --dataset-name random \
  --random-input-len 10000 \
  --random-output-len 1000 \
  --num-prompts 4 \
  --seed 1 \
  --temperature 1.0 \
  --request-rate inf \
  --max-concurrency 1 \
  --ignore-eos 2>&1 | tee bench.log

============ Serving Benchmark Result ============
Successful requests:                     4
Failed requests:                         0
Maximum request concurrency:             1
Benchmark duration (s):                  127.83
Total input tokens:                      40000
Total generated tokens:                  4000
Request throughput (req/s):              0.03
Output token throughput (tok/s):         31.29
Peak output token throughput (tok/s):    39.00
Peak concurrent requests:                2.00
Total token throughput (tok/s):          344.21
---------------Time to First Token----------------
Mean TTFT (ms):                          6112.72
Median TTFT (ms):                        6079.31
P99 TTFT (ms):                           6334.50
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          25.87
Median TPOT (ms):                        25.88
P99 TPOT (ms):                           25.91
---------------Inter-token Latency----------------
Mean ITL (ms):                           25.88
Median ITL (ms):                         25.91
P99 ITL (ms):                            26.93
==================================================

• Token Generation speed: 1 tok / 25.87ms = 38.65 tok/s
• Prompt Processing speed: 10k tok / 6112s = 1636 tok/s

【Model Files】

【Overview】

GLM-5.2

<div align="center"> <img src=https://raw.githubusercontent.com/zai-org/GLM-5/refs/heads/main/resources/logo.svg width="15%"/> </div> <p align="center"> 👋 Join our <a href="https://raw.githubusercontent.com/zai-org/GLM-5/refs/heads/main/resources/wechat.png" target="_blank">WeChat</a> or <a href="https://discord.gg/QR7SARHRxK" target="_blank">Discord</a> community. <br> 📖 Check out the GLM-5.2 <a href="https://z.ai/blog/glm-5.2" target="_blank">blog</a> and GLM-5 <a href="https://arxiv.org/abs/2602.15763" target="_blank">Technical report</a>. <br> 📍 Use GLM-5.2 API services on <a href="https://docs.z.ai/guides/llm/glm-5.2">Z.ai API Platform. </a> <br> 🔜 Try GLM-5.2 <a href="https://chat.z.ai">here</a>. </p> <p align="center"> [<a href="https://huggingface.co/papers/2602.15763" target="_blank">Paper</a>] [<a href="https://github.com/zai-org/GLM-5" target="_blank">GitHub</a>] </p>

Introduction

We're introducing GLM-5.2, our latest flagship model for long-horizon tasks. It marks a substantial leap in long-horizon task capability over its predecessor GLM-5.1 and, for the first time, delivers that capability on a solid 1M-token context. GLM-5.2's new capabilities include:

• Solid 1M Context: A solid 1M-token context that stably sustains long-horizon work
• Advanced Coding with Flexible Effort: Stronger coding capabilities with multiple thinking effort levels to balance performance and latency
• Improved Architecture: We propose IndexShare, which reuses the same indexer across every four sparse attention layers, reducing per-token FLOPs by 2.9× at a 1M context length. We also improve GLM-5.2’s MTP layer for speculative decoding, increasing the acceptance length by up to 20%
• Pure Open: An MIT open-source license — no regional limits, technical access without borders

Benchmark

Serve GLM-5.2 Locally

GLM-5.2 supports deployment with the following frameworks. Feel free to try them out:

• SGLang (v0.5.13.post1+) — see cookbook
• vLLM (v0.23.0+) — see recipes
• Transformers (v0.5.12+) — see transformers docs
• KTransformers (v0.5.12+) — see tutorial
• Unsloth (v0.1.47-beta+) — see guide
• For deployment on the Ascend NPU platform, inference frameworks such as vLLM-Ascend, xLLM and SGLang are supported — see here.

Footnote

• Humanity’s Last Exam (HLE) & other reasoning tasks: We use sampling parameters of temperature=1.0, top_p=0.95 for evaluation. We evaluate with a maximum generation length of 163,840 tokens. By default, we report the text-only subset; results marked with * are from the full set. For AIME, HMMT and IMOAnswerBench, we evaluate each question using the following system prompt: Your response should be in the following format:\nExplanation: {your explanation for your final answer}\nExact Answer: {your succinct, final answer}\nConfidence: {your confidence score between 0% and 100% for your answer}. We use GPT-5.5 (medium) as the judge model. For HLE-with-tools, we use a maximum context length of 300,000 tokens, with no context management strategy.
• SWE-Bench Pro: We run the SWE-Bench Pro suite with OpenHands using a tailored instruction prompt. Settings: temperature=1, top_p=1, max_new_tokens=32k, with a 400K context window.
• NL2Repo: We evaluated NL2Repo with temperature=1.0, top_p=1.0, and max_new_tokens=48k under 400k context. To prevent hacking, we use rule-based and a LLM-based judgement to prevent malicious behaviors (e.g., unauthorized pip or curl operations).
• DeepSWE: We run DeepSWE with the official pier evaluation framework and the mini-swe-agent harness (temperature=1.0, top_p=1.0, timeout=2h, 400K context). Each task is solved in an isolated container with 2 CPUs, 8 GB RAM, and no internet access.
• ProgramBench: We evaluate ProgramBench (200 instances) with Claude-Code 2.1.156 using temperature=1.0, top_p=1.0, max_tokens=64000, max_turns=2000, sample_timeout=6h, reasoning_effort=max, with a 400K context window. Each instance runs in a (4 CPUs, 8 GB RAM) sandbox with internet access disabled.
• Terminal-Bench 2.1 (Terminus 2): We evaluate Terminal-Bench 2.1 with Terminus-2 framework using parser=json, timeout=4h, temperature=1.0, top_p=1.0, max_new_tokens=48k, max_episodes=500, with a 256K context window. Resource limits are capped at 4 CPUs and 8 GB RAM.
• Terminal-Bench 2.1 (Claude Code): We evaluate in Claude Code 2.1.167 with temperature=1.0, top_p=0.95, max_new_tokens=131072. We override max_new_tokens to 128k via a transparent proxy, bypassing the 64k CLI cap to restore the configurability of CLAUDE_CODE_MAX_OUTPUT_TOKENS. We remove wall-clock time limits, while preserving per-task CPU and memory constraints. Scores are averaged over 5 runs.
• MCP-Atlas: All models were evaluated in think mode on the 500-task public subset with a 10-minute timeout per task. We use Gemini-3.0-Pro as the judge model for evaluation.
• Tool-Decathlon: We use the official evaluation service and set max_token to 128K.
• FrontierSWE: The evaluation was conducted by Proximal with 1M context length, max effort level, and 128K maximum output tokens. Dominance score reported as of 2026/06/16.
• PostTrainBench: The evaluation was conducted by PostTrainBench with 1M context length, max effort level, and 128K maximum output tokens.
• SWE-Marathon: The evaluation was conducted by Abundant AI with 1M context length, max effort level, and 128K maximum output tokens.

Citation

If you find GLM-5.2 useful in your research, please cite our technical report:

@misc{glm5team2026glm5vibecodingagentic,
      title={GLM-5: from Vibe Coding to Agentic Engineering},
      author={GLM-5-Team and : and Aohan Zeng and Xin Lv and Zhenyu Hou and Zhengxiao Du and Qinkai Zheng and Bin Chen and Da Yin and Chendi Ge and Chenghua Huang and Chengxing Xie and Chenzheng Zhu and Congfeng Yin and Cunxiang Wang and Gengzheng Pan and Hao Zeng and Haoke Zhang and Haoran Wang and Huilong Chen and Jiajie Zhang and Jian Jiao and Jiaqi Guo and Jingsen Wang and Jingzhao Du and Jinzhu Wu and Kedong Wang and Lei Li and Lin Fan and Lucen Zhong and Mingdao Liu and Mingming Zhao and Pengfan Du and Qian Dong and Rui Lu and Shuang-Li and Shulin Cao and Song Liu and Ting Jiang and Xiaodong Chen and Xiaohan Zhang and Xuancheng Huang and Xuezhen Dong and Yabo Xu and Yao Wei and Yifan An and Yilin Niu and Yitong Zhu and Yuanhao Wen and Yukuo Cen and Yushi Bai and Zhongpei Qiao and Zihan Wang and Zikang Wang and Zilin Zhu and Ziqiang Liu and Zixuan Li and Bojie Wang and Bosi Wen and Can Huang and Changpeng Cai and Chao Yu and Chen Li and Chengwei Hu and Chenhui Zhang and Dan Zhang and Daoyan Lin and Dayong Yang and Di Wang and Ding Ai and Erle Zhu and Fangzhou Yi and Feiyu Chen and Guohong Wen and Hailong Sun and Haisha Zhao and Haiyi Hu and Hanchen Zhang and Hanrui Liu and Hanyu Zhang and Hao Peng and Hao Tai and Haobo Zhang and He Liu and Hongwei Wang and Hongxi Yan and Hongyu Ge and Huan Liu and Huanpeng Chu and Jia'ni Zhao and Jiachen Wang and Jiajing Zhao and Jiamin Ren and Jiapeng Wang and Jiaxin Zhang and Jiayi Gui and Jiayue Zhao and Jijie Li and Jing An and Jing Li and Jingwei Yuan and Jinhua Du and Jinxin Liu and Junkai Zhi and Junwen Duan and Kaiyue Zhou and Kangjian Wei and Ke Wang and Keyun Luo and Laiqiang Zhang and Leigang Sha and Liang Xu and Lindong Wu and Lintao Ding and Lu Chen and Minghao Li and Nianyi Lin and Pan Ta and Qiang Zou and Rongjun Song and Ruiqi Yang and Shangqing Tu and Shangtong Yang and Shaoxiang Wu and Shengyan Zhang and Shijie Li and Shuang Li and Shuyi Fan and Wei Qin and Wei Tian and Weining Zhang and Wenbo Yu and Wenjie Liang and Xiang Kuang and Xiangmeng Cheng and Xiangyang Li and Xiaoquan Yan and Xiaowei Hu and Xiaoying Ling and Xing Fan and Xingye Xia and Xinyuan Zhang and Xinze Zhang and Xirui Pan and Xu Zou and Xunkai Zhang and Yadi Liu and Yandong Wu and Yanfu Li and Yidong Wang and Yifan Zhu and Yijun Tan and Yilin Zhou and Yiming Pan and Ying Zhang and Yinpei Su and Yipeng Geng and Yong Yan and Yonglin Tan and Yuean Bi and Yuhan Shen and Yuhao Yang and Yujiang Li and Yunan Liu and Yunqing Wang and Yuntao Li and Yurong Wu and Yutao Zhang and Yuxi Duan and Yuxuan Zhang and Zezhen Liu and Zhengtao Jiang and Zhenhe Yan and Zheyu Zhang and Zhixiang Wei and Zhuo Chen and Zhuoer Feng and Zijun Yao and Ziwei Chai and Ziyuan Wang and Zuzhou Zhang and Bin Xu and Minlie Huang and Hongning Wang and Juanzi Li and Yuxiao Dong and Jie Tang},
      year={2026},
      eprint={2602.15763},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2602.15763},
}