GLM-4.7-GGUF

This repo contains specialized MoE-quants for GLM-4.7. The idea being that given the huge size of the FFN tensors compared to the rest of the tensors in the model, it should be possible to achieve a better quality while keeping the overall size of the entire model smaller compared to a similar naive quantization. To that end, the quantization type default is kept in high quality (Q8_0 to Q5_K) and the FFN UP + FFN GATE tensors are quanted down along with the FFN DOWN tensors.

The mixture convention is as follows: [Default Type]-[FFN_UP]-[FFN_GATE]-[FFN_DOWN], eg: Q8_0-Q4_K-Q4_K-Q5_K. This means:

• Q8_0 is the default type (attention, shared expert, etc.)
• Q4_K was used for the FFN_UP and FFN_GATE conditional expert tensors
• Q5_K was used for the FFN_DOWN conditional expert tensors

I've mapped these mixes to the closest BPW I could reasonably discern.

Shout Outs

• Bartowski's imatrix_v3 corpus
• Ubergarm's ik_llama quants (for SOTA quant schemas)