Downloads · 30 days
82
23% of all-time downloads
schoggie/K2-Horizon-MoVA-36B-A4B-W4A16
K2-Horizon-MoVA-36B-A4B-W4A16 is a text generation model from schoggie. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
INT4 group-128 symmetric weights on the routed MoE experts only, produced with compressed-tensors (pack-quantized). Everything else — most importantly the entire MoVA mechanism — stays BF16.
Downloads · 30 days
82
23% of all-time downloads
All-time downloads
354
Public
Parameters
37.4B
35.5 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors35.5 GB · 100%
How the weights are stored.
I3226.5B · 71%
From the Hugging Face model README
INT4 group-128 symmetric weights on the routed MoE experts only, produced with
compressed-tensors (pack-quantized). Everything else — most importantly the entire MoVA
mechanism — stays BF16.
HumanEval pass@1: 92.1% (151/164, greedy) on a single NVIDIA CMP 170HX.
K2-Horizon's attention replaces v_proj with MoVA — a router (v_router) over 64
value experts per layer. A widely-used community GPTQ Int4 of this model quantizes those
value experts and the router to 4 bits. On our hardware that produced systematically
corrupted output: fluent English and correct reasoning, but broken code —
n = len(numbers))
for i in range(n)):
Token-level inspection showed the model emitting '))))\n\n' as a single token where
'))' was correct, i.e. picking the wrong member of the ) / )) / ))) / )))) family.
That is the signature of slightly-off logits, not a text bug. Measured HumanEval 15.2%,
with 106/164 samples containing the doubled-paren pattern. It reproduced identically with
CUDA graphs off (15.9%) and with an unquantized KV cache (16.5%), so it was neither.
The cause is that an INT4 router selects slightly wrong value experts, so the attention value vectors are subtly wrong at every layer.
IFM's own FP8 release says the same thing in its config. Its ignored_layers (3408
entries) leave in full precision: all 64 self_attn.v_experts.* per layer, v_router,
q/k/o/gate_proj, mlp.gate, mlp.shared_experts.*, the dense mlp_only_layers, every
norm, lm_head, embed_tokens, model.norm. IFM quantizes only mlp.experts.*.
This checkpoint follows that contract exactly: 13,500 tensors quantized (45 MoE layers × 100 experts × {gate,up,down}_proj), 3,498 copied in BF16. It is therefore larger than the GPTQ (35.5 GB vs 21 GB) — that is the point.
| GPTQ Int4 (MoVA quantized) | this W4A16 (MoVA BF16) | |
|---|---|---|
| HumanEval pass@1 | 15.2% | 92.1% |
| doubled-paren samples | 106/164 | none observed |
| size | 21 GB | 35.5 GB |
compressed-tensors W4A16. Note the ecosystem situation at time of writing:
xllm.py, the native K2 path) refuses any quantized K2 —
_XllmMoVAAttentionBase.__init__ raises "K2 Horizon MoVA supports unquantized bf16/fp16 weights only" whenever quant_config is not None. It serves K2 in BF16 only,
which needs ~70 GB (TP≥2).self.v_experts_fused.qweight — so this checkpoint fails to load with
AttributeError: 'MergedColumnParallelLinear' object has no attribute 'qweight'.A small patch adds a BF16 MoVA path (K2_MOVA_BF16=1): build v_experts_fused with
quant_config=None, then compute the value projection densely and gather the top-k experts.
It is mathematically identical to the sparse path (silu is elementwise per expert). With
E·kv_dim = 65,536 and hidden = 2560 that is ~0.13 GB of activations per 1000 tokens.
podman run -d --device nvidia.com/gpu=0 \
-v /path/to/models:/models -p 8000:8080 \
-e K2_MOVA_BF16=1 \
<vllm-image-with-patched-plugin> \
--model /models/k2-horizon-w4a16 --trust-remote-code \
--max-model-len 131072 --max-num-seqs 32 \
--gpu-memory-utilization 0.93 \
--attention-backend FLASHINFER --kv-cache-dtype fp8 --enable-prefix-caching
Weights 33.6 GiB resident. With --kv-cache-dtype fp8: 266,016-token KV pool,
131,072 per request (max_position_embeddings is 524,288).
| np | 1 | 2 | 4 | 8 | 16 | 32 |
|---|---|---|---|---|---|---|
| aggregate tok/s | 42.5 | 73.1 | 137.0 | 255.0 | 498.0 | 874.7 |
Single-stream is ~42% below the (corrupt) sparse-Marlin build because the dense fallback evaluates all 64 value experts rather than the top-4. A sparse BF16 gather path would recover most of that, but is data-dependent and therefore not CUDA-graph safe.
KV is expensive on this model: full attention on all 48 layers, 8 KV heads × 128 head_dim → ~192 KB/token BF16, ~96 KB/token FP8. Roughly 6× a hybrid-attention model of similar size.
Base model and the MoVA architecture: IFM/K2-Horizon-MoVA-36B-A4B by MBZUAI's Institute of Foundation Models. This repository is only a quantization.