K2-Horizon-3.7B-ROCmFP4-GGUF

K2-Horizon-3.7B — ROCmFP4 / ROCmFPX GGUF

Quantized GGUFs of IFM/K2-Horizon-3.7B for AMD ROCm, converted and measured on Strix Halo (gfx1151, ROCm 7.2.4).

K2-Horizon is not supported by upstream llama.cpp (tracked as ggml-org/llama.cpp#28361). These files were produced with our own k2-horizon architecture implementation; the conversion, the compute graph and the BPE pre-tokenizer are ours.

Files

Perplexity: 400 KB of English prose, -c 512 --chunks 20, full GPU offload, same binary and session for every row. BF16 reference for this model: 15.777.

STRIX_LEAN is built with --output-tensor-type q6_K (protected head), not the raw tier.

⭐ Q8_0_ROCMFPX_AGENT scores better than plain Q8_0_ROCMFPX at the same nominal 8.25 bpw, on this model and on its sibling. There is no MTP head in play here.

Architecture

Dense decoder, 36 layers, GQA 32 heads / 8 kv, head_dim 128, SwiGLU, RoPE theta 1e7, vocab 250624, declared context 524288. Hidden 2560, FFN 10240.

The distinguishing feature is grouped RMS normalisation: the norm is computed within attention.group_norm_groups = 2 contiguous groups along the hidden dimension rather than across the whole vector, at all three norm sites including the final norm. The norm weights are full width either way, so the grouping is not visible in the checkpoint.

⚠️ These files are not interchangeable with the vendor's GGUFs

IFM/K2-Horizon-3.7B-GGUF stores q/k in unpermuted HF layout, which its own llama.cpp fork rotates accordingly. These files carry the permute that llama.cpp's LLAMA_ROPE_TYPE_NORM expects.

Measured on the same corpus, binary and session, BF16 in both cases:

Use each with the build it was made for.

⛔ Context

The declared 524288 context allocates 72 GiB of KV (77,309,411,328 bytes) and fails on 128 GiB hardware. Pass -c explicitly; 131072 costs 19.3 GB and is what we serve.

⛔ Chat template

The model repo's chat_template.jinja uses {%- if spec is sameas true -%}. llama.cpp's jinja engine (minja) does not implement the sameas test, so a GGUF carrying that template fails at common_chat_templates_init with Parser Error: Expected %} (Got true).

These files embed the template from the vendor's own GGUF, which minja parses. Tool calling works with --jinja (verified 3/3). Reasoning level is a template input: reasoning_effort high → <ifm|think>, medium → <ifm|think_fast>, low → <ifm|think_faster>.

Tokenizer

tokenizer.ggml.pre = k2-horizon. The splitter differs from llama3's in that the contraction group is case-insensitive and the letter run also admits \p{M}, U+200C (ZWNJ) and U+200D (ZWJ).

Against HF tokenizers on a 50-case battery it matches 48/50. The two differences are combining-mark inputs where HF applies its declared NFC normalizer and llama.cpp's BPE path does not; the vendor's own GGUF produces the identical two differences.

Reproduction

git clone -b k2-horizon https://github.com/kingjones30/ROCmFPX.git && cd ROCmFPX
export ROCM_PATH=/opt/rocm HIP_PATH=/opt/rocm
cmake -S . -B build-rocm -DCMAKE_BUILD_TYPE=Release \
  -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DAMDGPU_TARGETS=gfx1151 \
  -DCMAKE_HIP_COMPILER=/opt/rocm/lib/llvm/bin/clang++ \
  -DCMAKE_HIP_COMPILER_ROCM_ROOT=/opt/rocm \
  -DGGML_HIP_GRAPHS=ON -DGGML_HIP_NO_VMM=ON -DLLAMA_CURL=OFF -DLLAMA_BUILD_WEBUI=OFF
cmake --build build-rocm -j12 --target llama-server llama-quantize llama-perplexity

hf download IFM/K2-Horizon-3.7B --local-dir ./src --exclude "*.png"
python3 convert_hf_to_gguf.py ./src --outtype bf16 --outfile ./K2-Horizon-3.7B-BF16.gguf
PYTHONPATH=gguf-py python3 gguf-py/gguf/scripts/gguf_new_metadata.py \
  ./K2-Horizon-3.7B-BF16.gguf ./tmp.gguf --chat-template-file models/templates/k2-horizon.jinja
mv ./tmp.gguf ./K2-Horizon-3.7B-BF16.gguf

build-rocm/bin/llama-quantize --output-tensor-type q6_K \
  ./K2-Horizon-3.7B-BF16.gguf ./K2-Horizon-3.7B-Q4_0_ROCMFP4_STRIX_LEAN.gguf Q4_0_ROCMFP4_STRIX_LEAN

build-rocm/bin/llama-server -m ./K2-Horizon-3.7B-Q4_0_ROCMFP4_STRIX_LEAN.gguf \
  -ngl 999 -fa on -fit off -dev ROCm0 --ctx-size 131072 --jinja

⛔ Plain -DGGML_HIP=ON fails with "Failed to find ROCm root directory" — pass CMAKE_HIP_COMPILER explicitly, as above.

Licence

Apache-2.0, inherited from IFM/K2-Horizon-3.7B.