GLM-5.2-504B-Code-GGUF

Model Description

This is a quantized version of the GLM-5.2-REAP-504B-GGUF model. The model has been converted from BF16 to GGUF format with importance matrix optimization.

Provided Quants

Other Files

Run it

llama-cli -m GLM-5.2-504B-Code-IQ2_XXS.gguf -p "Hello"
llama-server -m GLM-5.2-504B-Code-IQ2_XXS.gguf --host 0.0.0.0 --port 8080

Recommended serving — recover most of the loop gap for free

• Anti-loop (recommended):

  • min_p=0.05, repetition_penalty=1.05
  • min_p=0.05, repetition_penalty=1.10

  Start at 1.05; go to 1.10 if you see loops — a higher repetition penalty trades a little risk of over-penalizing legitimate repetition (e.g. in code) for near-zero looping.

• Conciseness: a brevity system prompt — "Be concise. Think only as much as the task needs, then answer and stop." — halves median length (1267 → 507 tokens). Note it does not reduce looping (that's the sampler's job); combine the two for short, low-loop output.

Quantization Details

This model has been quantized using llama.cpp's llama-quantize tool with importance matrix (--imatrix) for optimal quantization. The quantization target was IQ2_XXS (2-bit quantization).

Quantization Summary

Tensor Type to Quantization Method Mapping

Tensors Kept in Original Format (Not Quantized)

f32 (Float 32-bit) - Kept for Numerical Stability

• All *_norm.weight and *_norm.bias tensors
• All *_gate_inp.weight tensors (e.g., ffn_gate_inp.weight)
• All exp_probs_b.bias tensors
• blk.78.nextn.enorm.weight
• blk.78.nextn.hnorm.weight
• blk.78.nextn.shared_head_norm.weight

bf16 (bfloat16) - Not Quantized (Missing Importance Matrix Data)

• All indexer.attn_k.weight tensors
• All indexer.attn_q_b.weight tensors
• All indexer.proj.weight tensors
• blk.78.attn_kv_a_mqa.weight
• blk.78.attn_output.weight
• blk.78.attn_q_a.weight
• blk.78.attn_q_b.weight
• blk.78.attn_v_b.weight
• blk.78.ffn_down_exps.weight
• blk.78.ffn_down_shexp.weight
• blk.78.ffn_gate_exps.weight
• blk.78.ffn_gate_shexp.weight
• blk.78.ffn_up_exps.weight
• blk.78.ffn_up_shexp.weight
• blk.78.nextn.eh_proj.weight

Quantization Method Details

Important Notes

1. Fallback Quantization: 79 tensors required fallback quantization because:

   • attn_k_b.weight tensors have ncols=192 (not divisible by 256, required for IQ2_XXS), falling back to IQ4_NL
   • Other tensors missing importance matrix data

2. Indexer Tensors: All indexer.* tensors were kept in BF16 format as they are critical for the model's routing/indexing mechanism and no importance data was available.

3. Final Layer: blk.78 (the final transformer block) was largely left unquantized to preserve output quality, as importance data was missing for these tensors.

4. Dense Layers: The first three blocks (blk.0 to blk.2) use dense FFN layers instead of MoE, which were quantized with Q2_K.

5. Importance Matrix: Quantization used an importance matrix from 80 chunks of calibration data, optimizing which tensors receive more bits based on their importance to model performance.