GLM-5.2-AutoRound-W4G64-MTP

GLM-5.2 AutoRound W4G64

This is an unofficial community quantization of zai-org/GLM-5.2. It is not an official Z.ai release and is not affiliated with Z.ai.

The checkpoint targets high-throughput reasoning and coding-agent serving on a single 8-GPU Hopper node. Routed MoE expert weights use symmetric 4-bit AutoRound quantization with group size 64. Attention, the DSA/IndexShare indexer, shared experts, sensitive projections, and the output head remain in BF16. The MTP layer is included and can be used for speculative decoding.

Checkpoint summary

Quantization procedure

This is post-training weight-only quantization; the model was not fine-tuned. The run used AutoRound 0.14.0 on four NVIDIA H20 GPUs with:

The calibration set consisted of packed coding-agent-style text samples. The calibration corpus is not distributed with this checkpoint.

The mixed-precision policy was intentionally conservative:

• Backbone layers 3–77: routed-expert gate_proj, up_proj, and down_proj matrices are W4G64.
• Backbone layers 0–2, all attention and indexer modules, shared experts, eh_proj, weights_proj, embeddings, and lm_head remain BF16.
• MTP layer 78 was processed separately: 768 routed-expert weights are W4G64; 23 non-expert tensors remain BF16. The MTP shared head reuses the BF16 target head at runtime.

Validated runtime

The tested software stack was:

apply_vllm_glm52_patches.py contains the source-level compatibility changes used for this checkpoint. It is strictly hash-pinned to the vLLM commit and Triton source above, stages all edits before installation, compiles and verifies every output, installs files atomically, and is idempotent. It refuses unknown or partially modified sources instead of attempting a fuzzy patch.

The patch covers:

1. GLM-5.2 sparse-indexer and missing-parameter guards, plus AutoRound routed expert and MTP checkpoint namespace compatibility.
2. Header-first safetensors filtering so MTP eager loading does not read every unrelated backbone shard payload.
3. CUDA Graph pre-capture warmup and real cubin generation through ptxas.
4. Explicit PyNCCL loader/device handling and correct CUDA-device binding for the CPU KV-cache pinning thread.

These changes affect loading and serving only; they do not modify model weights. Use a separate environment for this pinned runtime. A newer vLLM version may already contain equivalent fixes and should be validated independently.

vLLM serving example

The following is a public, minimal reproduction of the tested serving profile. It assumes that vLLM has already been built at the commit above and that ptxas is available on PATH. Replace the example paths and public model name.

export VENV=/path/to/vllm-venv
export MODEL_DIR=/path/to/GLM-5.2-AutoRound-W4G64

"$VENV/bin/python" "$MODEL_DIR/apply_vllm_glm52_patches.py" \
  --venv-root "$VENV" \
  --model-dir "$MODEL_DIR" \
  --apply

# Keep compilation and CUDA Graphs enabled. These settings match the validated
# single-node runtime; adapt NCCL transport selection to your topology.
export VLLM_DISABLE_COMPILE_CACHE=1
export VLLM_DISABLE_PYNCCL=0
export NCCL_CUMEM_ENABLE=0
export NCCL_CUMEM_HOST_ENABLE=0
export VLLM_ALLREDUCE_USE_SYMM_MEM=0
export VLLM_USE_NCCL_SYMM_MEM=0
export TRITON_STORE_BINARY_ONLY=0
export CUDA_MODULE_LOADING=EAGER

"$VENV/bin/vllm" serve "$MODEL_DIR" \
  --served-model-name glm-5.2-autoround-w4g64 \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 8 \
  --dtype auto \
  --max-model-len 130000 \
  --gpu-memory-utilization 0.94 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 8192 \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice \
  --chat-template-content-format string \
  --trust-remote-code \
  --safetensors-load-strategy eager \
  --distributed-timeout-seconds 1800 \
  --disable-custom-all-reduce \
  --enable-prefix-caching \
  --compilation-config.pass_config.fuse_allreduce_rms false \
  --speculative-config '{"method":"mtp"}' \
  --spec-tokens 2 \
  --kv-offloading-size 256 \
  --kv-offloading-backend native \
  --disable-hybrid-kv-cache-manager

The 256 GiB CPU KV offload setting is optional and requires sufficient pinned host memory. Reduce or remove it for systems without that capacity.

Example OpenAI-compatible request:

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "glm-5.2-autoround-w4g64",
    "messages": [{"role": "user", "content": "Explain and solve x^2 = 1 (mod 105)."}],
    "max_tokens": 8192,
    "temperature": 0.6,
    "top_p": 0.95,
    "stream": true
  }'

Reasoning models may use several thousand tokens before final content. For long-horizon coding-agent workloads, a 32K output allowance was materially more reliable than an 8K allowance.

Measured performance

Measurements below used the validated stack on one 8 × H20 node with TP8, 130K maximum context, MTP speculative depth 2, max_num_seqs=8, an 8,192-token batch ceiling, prefix caching, CUDA Graphs, and 256 GiB native CPU KV offload.

The fixed-shape serving test used the same 117-token prompt, exactly 512 output tokens per request, streaming responses, and a 60-second load window. Throughput is the server generation-token counter divided by wall time.

No request preemption or metrics scrape error occurred in these windows. MTP accepted 57.7%–58.7% of drafted tokens, for an effective accepted length of about 2.16 tokens per verification iteration.

A separate reasoning smoke test produced 4,083 tokens at 110.16 tok/s after a 0.266-second TTFT, stopped normally, returned both reasoning and final content, and correctly enumerated the eight solutions of x² ≡ 1 (mod 105).

On isolated coding-agent tasks, the model passed all public and held-out tests for the four completed single-concurrency tasks, ranging from configuration merging to a parser/formatter/evaluator. In concurrent agent runs, aggregate generation was 218.20 tok/s at concurrency 3 and 278.04–292.79 tok/s at concurrency 5. These are workload observations, not standardized leaderboard results.

Performance varies with prompt length, output distribution, MTP acceptance, cache state, host-memory bandwidth, software build, and GPU topology. The numbers above should not be compared with other model cards unless the harness and serving configuration are matched.

Quality and limitations

• This checkpoint has not been evaluated against the BF16 base model with a comprehensive, identical public benchmark harness. No claim of lossless quantization is made.
• Only the vLLM commit and 8 × H20 configuration documented above were fully validated. Other vLLM revisions, runtimes, GPU architectures, and tensor parallel sizes may require additional work.
• Quantization is concentrated in routed experts, but quality can still differ from the BF16 base model, especially on rare domains or long reasoning paths.
• The checkpoint inherits the capabilities, risks, usage restrictions, and limitations described in the official GLM-5.2 model card.
• Users are responsible for application-specific safety, privacy, reliability, and output-quality evaluation before deployment.

License and references

This quantized checkpoint follows the base model's MIT license. Review the base repository and license before redistribution or production use.

• Official GLM-5.2 model card
• GLM-5 technical report
• AutoRound project
• AutoRound paper
• vLLM