K2-Horizon-3.7B-GGUF

K2-Horizon-3.7B — GGUF Quantizations

GGUF quantizations of IFM/K2-Horizon-3.7B, a dense 3.7B-parameter causal decoder (K2HorizonForCausalLM, model_type: k2_horizon).

Quantized by NANI-Nithin using a custom pipeline built on the MBZUAI-IFM llama.cpp fork (branch model/K2Horizon).

Note on file naming: IFM's official GGUF repo ships this model as K2-Horizon-4B-BF16.gguf (rounded name). Files in this repo are renamed to the K2-Horizon-3.7B-* convention for consistency.

---

Model Details

Note: These GGUFs carry the k2-horizon architecture token and require the MBZUAI-IFM fork (or upstream llama.cpp once support is merged) to run. Vanilla upstream llama.cpp (as of September 2026) does not support K2HorizonForCausalLM.

---

Included Files

Standard Quantizations

IQ (Importance-Matrix) Quantizations

---

Quantization Method

• Source: IFM's official BF16 GGUF (originally named K2-Horizon-4B-BF16.gguf, renamed here).
• imatrix: Computed from Salesforce/wikitext (wikitext-2-raw-v1, 500 rows) with 24 GPU layers offloaded on an RTX 4060 Laptop (8 GB VRAM). Applied to all K-quants below Q6 and all IQ quants.
• Fork: MBZUAI-IFM/llama.cpp, branch model/K2Horizon.

---

Usage

Requires the MBZUAI-IFM llama.cpp fork (model/K2Horizon branch).

git clone -b model/K2Horizon https://github.com/MBZUAI-IFM/llama.cpp
cd llama.cpp && cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release

./build/bin/llama-cli \
  -m K2-Horizon-3.7B-Q4_K_M.gguf \
  -p "Hello, I am" \
  -n 128

---

License

Weights are released under the same license as the original IFM/K2-Horizon-3.7B model. Please refer to the original repository for full license terms.

---

Credits

• Original model: MBZUAI IFM
• Quantized by: NANI-Nithin
• Quantization tool: MBZUAI-IFM/llama.cpp