GLM-5.2-GGUF-2.244bpw

GLM-5.2-GGUF-2.244bpw

This is a 2.2 BPW quantized model for the GPU riches with more combined RAM + VRAM than common sense.

The quant aims to achieve best-in-class performance, by relying on SOTA quants from ik_llama.cpp:

• Routed experts tensors use the IQ2_KT quant (2.125 BPW)
• Indexer tensors use either the Q6_0 quant (6.5 BPW) or the Q8_0 quant (8.5 BPW)
• All other tensors use the Q6_0 quant

Coupled with the recent enhancements:

• MTP support with e.g. --spec-type mtp:n_max=4,p_min=0.0 (#1890)
• graph parallel support with -sm graph (#1821)
• DSA support with -dsa -fidx (#2045, #2098, #2109, and many others)

it should run at decent speed as well, with very little slowdown at long context.

(Note: For now, -sm graph and quantize KV cache e.g. -ctk q8_0 do not work together with -dsa -fidx. As a fun exercise, you can ask this quant to get them to work.)

Versions

There are 3 versions:

• GLM-5.2-GGUF-2.244bpw.gguf - Made with the imatrix from unsloth (thanks!)
• GLM-5.2-GGUF-2.244bpw-q8indexer.gguf - Same as the above, but with Q8_0 for the indexer tensors
• GLM-5.2-GGUF-2.244bpw-muzzy-imatrix.gguf - Same as the above, but with the imatrix from muzzy (thanks!)

Comparison:

The PPL results were gathered using llama-perplexity with -f wiki.test.raw -c 512.

Recommendation: GLM-5.2-GGUF-2.244bpw-muzzy-imatrix.gguf - The imatrix from muzzy, which took him 5 days to generate, is just amazing.

From testing, all 3 versions work very well for agentic coding.

Size

Size from llama-server output:

llm_load_print_meta: model size       = 196.756 GiB (2.244 BPW)
llm_load_print_meta: repeating layers = 195.315 GiB (2.233 BPW, 751.427 B parameters)

Buffer size with -cmoe --no-mmap:

llm_load_tensors:        CPU buffer size = 185087.34 MiB
llm_load_tensors:      CUDA0 buffer size = 13724.12 MiB

<details> <summary>Recipe</summary>

# Attention
blk\..*\.attn_k_b\.weight=q6_0
blk\..*\.attn_v_b\.weight=q6_0

blk\..*\.attn_kv_a_mqa\.weight=q6_0
blk\..*\.attn_q_a\.weight=q6_0
blk\..*\.attn_q_b\.weight=q6_0
blk\..*\.attn_output\.weight=q6_0

# First 3 Dense Layers
blk\..*\.ffn_down\.weight=q6_0
blk\..*\.ffn_(gate|up)\.weight=q6_0

# Shared Expert Layers
blk\..*\.ffn_down_shexp\.weight=q6_0
blk\..*\.ffn_(gate|up)_shexp\.weight=q6_0

# Routed Experts Layers
blk\..*\.ffn_(up|gate|down)_exps\.weight=iq2_kt

# Indexer
blk\..*\.indexer\.proj\.weight=q6_0
blk\..*\.indexer\.attn_k\.weight=q6_0
blk\..*\.indexer\.attn_q_b\.weight=q6_0

# NextN MTP Layer
blk\..*\.nextn\.embed_tokens\.weight=q6_0
blk\..*\.nextn\.shared_head_head\.weight=q6_0
blk\..*\.nextn\.eh_proj\.weight=q6_0

# Non-Repeating Layers
token_embd\.weight=q6_0
output\.weight=q6_0

</details>

Flags

Example that uses DSA with IndexShare:

--no-mmap -ngl 99 --fit \
-mla 1 -dsa -fidx \
-c 102400 \
-b 2048 -ub 2048 -wgt 1 \
-cram 0 -muge -cuda graphs=1 \
--jinja --parallel-tool-calls \
--chat-template-kwargs {"reasoning_effort": "high"} \
--spec-type mtp:n_max=4,p_min=0.5

Speed comparison for tasks that can really benefit from MTP:

• without MTP:

prompt eval time =   22486.83 ms /  5188 tokens (    4.33 ms per token,   230.71 tokens per second)
       eval time =   54116.43 ms /   639 tokens (   84.69 ms per token,    11.81 tokens per second)
      total time =   76603.26 ms /  5827 tokens

• with MTP:

prompt eval time =   23404.00 ms /  5188 tokens (    4.51 ms per token,   221.67 tokens per second)
       eval time =   35463.34 ms /   639 tokens (   55.50 ms per token,    18.02 tokens per second)
      total time =   58867.34 ms /  5827 tokens
draft acceptance rate = 0.98259 (  508 accepted /   517 generated)
statistics mtp: #calls(b,g,a) = 1 130 130, #gen drafts = 130, #acc drafts = 129, #gen tokens = 517, #acc tokens = 508, dur(b,g,a) = 0.001, 1239.635, 0.063 ms

Comparison with other quants

The IQ2_KS from https://huggingface.co/muzzy/GLM-5.2-GGUF, using the same recipe as the smol-IQ2_KS from https://huggingface.co/ubergarm/GLM-5.1-GGUF, is 9 GiB larger, and has a slightly worse PPL. However, IQ2_KS can provide better PP and TG than IQ2_KT, especially on CPU. It is a trade-off between quality and speed.

On the other hand, the UD-IQ1_M from https://huggingface.co/unsloth/GLM-5.2-GGUF is 16 GiB larger, while having a much worse PPL. The UD-IQ2_XXS is 25 GiB larger, yet still having a slightly worse PPL. Unless one is stuck with the mainline llama.cpp, there is no good reason to use these unsloth quants.