GLM-4.6V-MP-GGUF

Mixed Precision GGUF layer quantization of GLM-4.6V by zai-org

Original model: https://huggingface.co/zai-org/GLM-4.6V

The hybrid quant employs different quantization levels on a per layer basis to enable both high performance and small file size at the same time. This quant is sized to be able to run on a 64G RAM CPU machine. The quants employed are all K to avoid slow CPU or older GPU processing of IQ quants. For this file the Q4_P_H layer quants are as follows:

Q4_K_L : Q4_K_M + attn_o = q6_k
Q5_K_L : attn_v = q8_0 attn_o = q6_k ffn_d = q6_k
Q6_K_S : Q6_K

   LAYER_TYPES='[
   [0 ,"Q4_K_M"], [1 ,"Q4_K_S"], [2 ,"Q4_K_S"], [3 ,"Q3_K_L"], [4 ,"Q3_K_L"], [5 ,"Q3_K_M"], [6 ,"Q3_K_L"], [7 ,"Q3_K_M"],
   [8 ,"Q3_K_L"], [9 ,"Q3_K_M"], [10,"Q3_K_M"], [11,"Q3_K_M"], [12,"Q3_K_M"], [13,"Q3_K_M"], [14,"Q3_K_M"], [15,"Q3_K_M"],
   [16,"Q3_K_M"], [17,"Q3_K_M"], [18,"Q3_K_M"], [19,"Q3_K_M"], [20,"Q3_K_M"], [21,"Q3_K_M"], [22,"Q3_K_M"], [23,"Q3_K_M"],
   [24,"Q3_K_L"], [25,"Q3_K_M"], [26,"Q3_K_L"], [27,"Q3_K_M"], [28,"Q3_K_L"], [29,"Q3_K_M"], [30,"Q3_K_L"], [31,"Q4_K_S"],
   [32,"Q4_K_S"], [33,"Q3_K_L"], [34,"Q4_K_S"], [35,"Q3_K_L"], [36,"Q4_K_S"], [37,"Q4_K_S"], [38,"Q4_K_S"], [39,"Q4_K_M"],
   [40,"Q4_K_M"], [41,"Q4_K_M"], [42,"Q4_K_L"], [43,"Q5_K_S"], [44,"Q5_K_M"], [45,"Q5_K_L"], [46,"Q6_K_S"]
   ]'

   FLAGS="--token-embedding-type Q4_K --output-tensor-type Q6_K --layer-types-high --tensor-pad [[10944,11008],[1408,1536]] --override-kv glm4moe.feed_forward_length=int:11008  --override-kv glm4moe.expert_feed_forward_length=int:1536

Model FFN length is padded from 10944 to 11008 to allow use of K quants in the layers instead of fallback legacy quants. Expert tensors are padded from 1466 to 1536 for the same reason.

Comparison:

The quant was sized to be able to run on 64G RAM CPUs along with the mmproj and then evaluated for acceptable reasoning performance across a curated set of text and vision test prompts.

Usage:

This is a RL trained (thinking) vision model. The layer quants for this model were evaluated on a set of test/eval prompts using greedy sampling. The model will fall into infinite generations on some prompts with greedy sampling. This effect can be mitigated by use of temperature sampling. Image mode was tested on a small set of images and found to be both functional and very accurate.

To run the model it is necessary to offload expert tensors to CPU:

OT="-ot exps=CPU -ngl 99"

This will open up a large amount of space in VRAM for context. Token gen rate is around 7tps on a 9900k / 4070 rig.

Benchmarks:

A limited set of vision benchmarks for the model will eventually be published here: https://huggingface.co/spaces/steampunque/benchlm

Download the file from below:

A discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:

https://github.com/ggml-org/llama.cpp/discussions/13040