K2-Horizon-MoVA-36B-A4B-mixed-NVFP4-FP8

<p align="center"> <img src="assets/banner.png" alt="Primitive — K2-Horizon-MoVA-36B-A4B mixed NVFP4/FP8" width="100%"> </p> <p align="center"> <img src="https://img.shields.io/badge/size-44.48%20GB-F98424?style=flat-square" alt="size 44.48 GB"> <img src="https://img.shields.io/badge/vs%20BF16-1.7×%20smaller-F98424?style=flat-square" alt="1.7x smaller than BF16"> <img src="https://img.shields.io/badge/vs%20official%20FP8-8%25%20smaller-F98424?style=flat-square" alt="8 percent smaller than the official FP8"> <img src="https://img.shields.io/badge/knowledge-level%20with%20BF16%20%26%20FP8-3DD13D?style=flat-square" alt="knowledge level with BF16 and FP8"> <img src="https://img.shields.io/badge/call%20accuracy-1.4%20below%20FP8-lightgrey?style=flat-square" alt="call accuracy 1.4 below the official FP8"> <img src="https://img.shields.io/badge/vs%20our%20NVFP4-%2B1.7%20call%20accuracy-F98424?style=flat-square" alt="plus 1.7 call accuracy versus our pure NVFP4 build"> <img src="https://img.shields.io/badge/requires-Blackwell-lightgrey?style=flat-square" alt="requires Blackwell"> <a href="https://primitive.com"><img src="https://img.shields.io/badge/web-primitive.com-lightgrey?style=flat-square" alt="primitive.com"></a> </p> <h3 align="center">8% smaller than the official FP8, level on knowledge, 1.4 behind on call accuracy.</h3> <p align="center"> Mixed-precision quantization of <a href="https://huggingface.co/IFM/K2-Horizon-MoVA-36B-A4B">IFM/K2-Horizon-MoVA-36B-A4B</a> to <b>44.48 GB</b>, on one 96 GB card.<br> 15 of the 45 sparse layers carry NVFP4 experts; the other 30 carry FP8 with block scales. </p>

This is the better of our two builds on call accuracy, and it needs Blackwell. There is also a pure NVFP4 build at 36.74 GB that is 7.74 GB smaller, 1.21× faster at concurrency 32, and the only one of the two that runs on Ampere. Take this one if tool calling matters most; take that one for pre-Blackwell hardware, the smallest footprint, or throughput under load.

---

Why this quant

• 🗜️ 8% smaller than IFM's own FP8, 1.7× smaller than BF16. 44.48 GB against 48.36 GB and 74.89 GB.
• 🎯 Knowledge is level with both references. 87.3 against the official FP8's 86.8 and BF16's 85.9 over 1,170 items. It is the highest number in that column and it is still a tie.
• 🧭 The better of our two builds on tool calls. 73.5 on call accuracy against the pure NVFP4 build's 71.8, for 7.74 GB more — the one axis that separates the two (t = +2.4). Against the official FP8 the gap is 1.4 and neither half of the suite separates on its own.
• 🧠 Block scales, not bit width, are what moved tool calling here. Two thirds of the expert layers carry FP8 with [128,128] block scales rather than group-wise 4-bit.
• 🖥️ One card, no tensor parallelism. Both official recipes use TP=2; this fits TP=1 with 231,488 tokens of KV at 32K context.
• ⚠️ Blackwell. Two thirds of the expert layers carry FP8 block scales and sm_80 has no FP8 tensor cores, so we neither built nor tested this one for Ampere. The pure NVFP4 build is verified there.
• 🔧 Serves on stock vLLM main, no patches and no custom kernels. Plain compressed-tensors.

---

Serve it

K2-Horizon support merged into vLLM main on 3 September 2026 (#55063) and is not in a release yet — 0.28.0 predates it. Use a main/nightly build:

hf download primitive-ai/K2-Horizon-MoVA-36B-A4B-mixed-NVFP4-FP8 \
  --local-dir ./K2-Horizon-MoVA-36B-A4B-mixed-NVFP4-FP8

docker run --gpus all --ipc=host -p 8000:8000 -v $PWD:/models \
  vllm/vllm-openai:nightly-8a728663c1c3eeace834a95f5654fa653cc1998c \
  --model /models/K2-Horizon-MoVA-36B-A4B-mixed-NVFP4-FP8 \
  --trust-remote-code --max-model-len 32768 \
  --gpu-memory-utilization 0.92 \
  --reasoning-parser k2_horizon

That image tag is the nightly every number on this page was measured on (vllm 0.28.1rc1.dev388). On one RTX PRO 6000 Blackwell it leaves 43.02 GiB resident for weights and 42.39 GiB for KV — 231,488 tokens at 32K context, 7.06× concurrency. Expect a slow first boot: engine init takes 198 s, of which 90 s is torch.compile, because the FP8 block-scale path compiles more kernels than a pure NVFP4 one (97 s and 3 s respectively for our other build).

Reasoning effort is a server-level choice on this architecture, not a per-request one. The template emits a different tag pair per effort and K2HorizonReasoningParser picks one pair once at startup, defaulting to high:

• high (the default) — <ifm|think> … </ifm|think>
• medium — <ifm|think_fast> … </ifm|think_fast>
• low — <ifm|think_faster> … </ifm|think_faster>

Send a per-request reasoning_effort: low against a server parsing high and the tags never match, so the whole reasoning trace stays in content instead of being separated out. Set the effort on the server. Split reasoning is returned in message.reasoning.

---

Measured

1,370 items across fourteen public benchmarks. A 1,170-item knowledge suite — MMLU-Pro, ARC-Challenge, HellaSwag, WinoGrande, CommonsenseQA, BoolQ, OpenBookQA, GSM8K, MATH-500 — and a 200-item tool-calling suite — BFCL v4, xLAM/APIGen, ToolACE, Glaive v2, When2Call — under one fixed protocol for every row: PQE-1.2, temperature 0.6 / top_p 0.95 / top_k 20, thinking forced on, a 16,384-token budget, concurrency 32, all rows back to back in one sitting on one RTX PRO 6000 Blackwell.

Auto-scored, no LLM judge. A knowledge item is scored on the last ANSWER: in the reply; a tool-calling item counts only if every expected call is present with acceptable arguments and no call is invented. Sizes are the safetensors weights, and ± is the sample standard deviation over the runs counted in the runs k/a column.

call is accuracy on the 160 tool-calling items that require a call; abstain is the 40 whose correct action is to call nothing; tool calling is the two pooled as 200. We report the halves separately because pooling hides which one moved. runs k/a is the number of knowledge and agentic runs behind each row. finished is the share of the 1,170 knowledge items that answered inside the token budget; out/answer is mean output tokens.

Knowledge is one group. This build's 87.3 is the highest number in the column and it does not separate from anything: against the official FP8 it is t = +0.6, at n=2 per side. Don't rank on this column.

Tool calling: where the 1.4 sits. The pooled 200-item number separates from the official FP8 (t = −2.6 over nine runs here and six there), but neither half does on its own: call is 1.4 low at t = −1.7 and abstain is 1.5 low at t = −1.0. So the loss is small, real in aggregate, and spread across both halves of the suite rather than concentrated in one. Our pure NVFP4 build behaves differently — there the loss is 3.1 on call alone (t = −4.2) and abstention is untouched.

It is not a formatting failure. Unparsed replies run at 14 per 1,800 here (0.78%) against 13 per 1,200 for the official FP8 (1.08%) and 6 per 600 for BF16 (1.00%) — cleaner than either IFM build, and our pure NVFP4 build is cleaner still at 0.75%. Thinking fired on 99–100% of items in every row.

<details> <summary>Per-benchmark detail, both suites</summary>

Pooled over every run behind the row, so a cell rests on runs × items. when2call supplies the 40 abstention items; the other four tool-calling benchmarks supply the 160 call items.

Where the tool-calling difference sits. xlam/APIGen is 40.2 here against the official FP8's 44.8, 4.6 points on 45 items and this build's largest gap against that reference. The pure NVFP4 build gives up 7.0 points on the same benchmark, so this build recovers about a third of that loss. xlam items carry the longest argument lists in the suite.

</details>

The noise floor, and why the run counts are in the table. Every build compared here has at least six agentic runs, the one exception being the BF16 reference at three. It needs to be six. The official FP8's tool-calling figure read 72.7 ±0.3 after three runs and 72.0 ±0.8 after six; this build's read 70.3 ±1.6 after three and 70.6 ±1.4 after nine. On the 1,170-item knowledge suite, repeat runs of a single checkpoint span up to 1.9 points, so treat a knowledge gap of a point or so as unresolved. The widest gap in that column is 1.4, between this build and BF16, and even that is only t = +1.6. We publish ties as ties.

Two footnotes on the table. The BF16 row was served at 20,480 context against the quants' 32,768, because 74.89 GB of weights leaves only 81,664 tokens of KV on a 96 GB card; the generation budget was the same 16,384 tokens for every row. And out/answer is a mean over a long tail: median output is 204 tokens against a mean of 1,118, a ratio of 5.5×. That shape usually means a model that reaches its answer and cannot stop. Here it does not — the official FP8 shows 1,176 against a median of 211 on the same items, a ratio of 5.6×, with the same ~4% of items hitting the budget. It is how K2-Horizon reasons, and quantization did not cause it.

---

Throughput

Cache-free, 8,192 tokens in and 512 out, one RTX PRO 6000 Blackwell, same nightly and same serve flags as the accuracy rows. Prefix caching is disabled and every measurement runs on its own seed, so no row is reading a cache the previous one filled. These come from one sitting and drift a few percent between sittings, so read them as a bracket. BF16 is absent because at 74.89 GB it only serves 20,480 context on this card, which would not be the same measurement.

Quantizing this model buys throughput under load, not single-stream latency. Those three builds span 11.6 GB and land within 0.71 tok/s of each other at concurrency 1. At concurrency 32 their per-token decode cost is nearly flat too — 89.2, 89.5 and 93.1 ms, a 4.4% spread — so the aggregate differences are not a decode-rate effect. They show up in prefill and in scheduling under load, where median time to first token falls from 4.46 s to 1.54 s across the range. Decode stays pinned because vLLM restacks the 15.1 GB of BF16 MoVA value-experts on every forward pass whatever the routed experts are quantized to, and that fixed cost dominates a single stream.

This build sits between the two on all five columns, which is what its size predicts. If throughput under load matters more than call accuracy, the pure NVFP4 build is 1.21× this one at concurrency 32 and reaches first token in 1.54 s against 3.59 s.

---

Choosing between our two builds

This build spends 7.74 GB, +21%, and call accuracy is the one axis that pays it back: +1.7 at t = +2.4. The pooled tool-calling lead of 1.1 does not separate (t = +1.5), nor does knowledge (t = +0.9). If tool calling dominates your workload and you have 48 GB, IFM's own FP8 is still the most accurate option — it leads this build by 1.4 on call accuracy. What this build offers against it is 8% less memory at a gap that neither half of the suite resolves on its own.

On Ampere. We have not tested this build on an A100 and do not expect FP8 block scales to load on sm_80, which has no FP8 tensor cores. The pure NVFP4 build does run there through Marlin dequantization, verified on an A100 40 GB: loading, serving and generating correctly with the reasoning parser working.

---

Comparable with our other models

Accuracy numbers move for reasons that have nothing to do with the model: a shorter token budget, a different temperature, or whether the model was allowed to reason at all. So every number in this table, on this card and on our other cards, comes from the one fixed protocol described above — the same 1,370 items, auto-scored, no LLM judge.

overall pools the two suites as 1,370 items, weighted 85.4% knowledge and 14.6% tool calling by item count. Read it with finished: overall scores an answer that overran the token budget as wrong, but it cannot tell you whether the model needed the room or failed to stop — finished and out/answer separate those, and this model's 96.3% is among the lowest finished in the table because it reasons in long chains. A gap under 1.0 in overall is a tie. Sizes are as each card reports them, which mixes GB and GiB.

---

What's quantized to what

The model is 37.4 B parameters in five very unequal pieces:

Which of those pieces got which format:

compressed-tensors, top-level format mixed-precision, two config groups. The NVFP4 group is W4A4 and carries observed activation scales plus a tensor-level weight_global_scale; the FP8 group is weight-only with block scales.

The second-largest block cannot be quantized at all. vLLM's MoVA attention does this on every forward pass:

w1 = torch.stack([expert.weight for expert in self.v_experts], dim=0).contiguous()
v  = fused_mova_impl(..., w1=w1, ...)

It stacks the raw .weight of all 64 value-experts and hands fused_mova_impl a dense tensor. A quantized Linear carries weight_packed plus scales and no weight, so a quantized v_expert either fails at load or feeds packed bytes to a kernel that reads them as bfloat16. IFM's own FP8 ignores all 2,880 of them for the same reason. That is 20% of the model pinned to BF16 in any build, which is why lower precision buys less here than the parameter count suggests: those 7.55 B parameters are 15.1 GB of BF16 that every forward pass moves regardless. Per the throughput section, that is why decode speed barely moves.

Two engine constraints shape the rest. MoVA fuses q and k into a single qk_proj, so those two would have to share one scheme. And a layer's experts are quantized as a unit: vLLM's fused MoE requires one scheme across an expert's gate/up/down, so the mixture here is per layer, never per projection.

---

Attribution

• Base: IFM/K2-Horizon-MoVA-36B-A4B, Apache-2.0, BF16 weights unmodified.
• Built with llm-compressor 0.13.1 and compressed-tensors 0.18.1.
• Calibration: 496 conversations at seqlen 2048, used to observe activation scales.
• The checkpoint carries IFM's modeling_k2_horizon.py and configuration_k2_horizon.py unchanged, so --trust-remote-code works for tooling that wants the reference implementation.
• Audited before publication: every fused NVFP4 group shares one weight_global_scale, every FP8 block-scale tensor has the shape its group declares, no activation scale is zero or NaN, and config.json agrees with the tensors actually present.

---

<p align="center"> <br> <img src="assets/primitive-logo.png" alt="Primitive" width="34"><br> _{<a href="https://primitive.com"><b>primitive</b></a> · <a href="https://huggingface.co/primitive-ai">more models</a> · inference economics for production LLM systems} </p>