Kimi-K2.7-Code-GGUF

imatrix Quantization of moonshotai/Kimi-K2.7

ik_llama.cpp quants of moonshotai/Kimi-K2.7 using Unsloth's imatrix and Ubergarm's quant recipes*. *embedding and output tensors left at q8_0

The other quants in this collection REQUIRE ik_llama.cpp fork to support the ik's latest SOTA quants and optimizations! Do not download these big files and expect them to run on mainline vanilla llama.cpp, ollama, LM Studio, KoboldCpp, etc! NOTE ik_llama.cpp can also run your existing GGUFs from AesSedai, unsloth, bartowski, mradermacher, etc

Some of ik's new quants are supported with Nexesenex/croco.cpp fork of KoboldCPP with Windows builds for CUDA 12.9. Also check for Windows builds by Thireus here. which have been CUDA 12.8.

These quants provide best in class perplexity for the given memory footprint.

The IQ2_KT is the most accurate 2-bit Kimi-K2.7-Code quant I've found on huggingface but it's slower to run.

Available quants

IQ2_KT - 264.5 GiB

Final estimate: PPL over 568 chunks for n_ctx=512 = 2.8960 +/- 0.01474 (+44.14% vs baseline)

IQ2_KS - 270.9 GiB

Final estimate: PPL over 568 chunks for n_ctx=512 = 2.9740 +/- 0.01518 (+48.02% vs baseline)

IQ2_KL - 329.7 GiB

Final estimate: PPL over 568 chunks for n_ctx=512 = 2.4417 +/- 0.01166 (+21.52% vs baseline)

IQ3_KT - 381.8 GiB

PPL Untested / don't have the hardware. Responds coherently to a few prompts.

References

• ik_llama.cpp
• Great mainline MoE optimizd quants AesSedai/Kimi-K2.7-GGUF

ACK

• Original Imatrix from Unsloth/Kimi-K2.7-Code-GGUF converted via https://gghfez-ik-llama-imatrix-converter.hf.space/
• Quant Recipes and parts of the README.md based off ubergarm/Kimi-K2.6-GGUF