K2-Horizon-MoVA-36B-A4B-EAGLE3

K2-Horizon-MoVA-36B-A4B — EAGLE-3 draft head

A 1-layer EAGLE-3 draft head for IFM/K2-Horizon-MoVA-36B-A4B — the sparse MoE frontier model with MoVA (Mixture-of-Values) attention. It provides lossless speculative decoding for the GPTQ-Int4 target served with vLLM on a single 48 GB GPU (L40S / A6000).

Community project, not affiliated with MBZUAI IFM or the vLLM team. Trained with vLLM Speculators.

What it is

• Architecture: Eagle3DraftModel (1 llama-style decoder layer + fusion fc), trained on auxiliary hidden states from target layers (2, 24, 45) plus the final hidden state as the distillation target.
• Reduced draft vocabulary — 32 768 (mapped back to the target's 250 624 vocab via d2t/t2d). This is what makes speculation net-positive here: a full-vocab draft head over 250 624 tokens costs almost as much per token as the 4 B-active target itself, and made decoding slower. The 32 K head is ~7.6× cheaper.
• Shares the target's tokenizer; lossless (greedy output is identical to the target).

Serving (vLLM + the k2-horizon-vllm plugin)

Requires the native vLLM plugin: github.com/stefanskiasan/k2-horizon-vllm (registers K2HorizonForCausalLM, exposes the EAGLE-3 auxiliary hidden states).

export VLLM_ATTENTION_BACKEND=TRITON_ATTN          # for int4 KV
vllm serve <K2-Horizon-GPTQ-Int4> \
  --trust-remote-code --tensor-parallel-size 1 \
  --kv-cache-dtype int4_per_token_head \
  --speculative-config '{"method":"eagle3","model":"<this-repo>","num_speculative_tokens":2}' \
  --served-model-name k2-horizon

num_speculative_tokens: 2 is the sweet spot on this target (acceptance is front-loaded; 3 wastes the tail).

Performance (NVIDIA L40S, 48 GB, GPTQ-Int4, single stream, greedy)

Measured against an identical no-spec baseline on the same server (70.1 tok/s, deterministic):

Speculation is lossless, so this is pure latency reduction. Acceptance is front-loaded (per-position ≈ 0.44 / 0.17), which is why num_spec=2 wins — a small 1-layer draft against a sparse MoE target. Structured / repetitive workloads (tool-calling, JSON, code) accept far more than the general-chat (UltraChat) prompts benchmarked here, so real-world agentic speedups are higher. Trained on only ~4 500 conversations; more data is the lever for higher acceptance.

Training

vLLM Speculators, 1 layer, 8 epochs, AdamW, lr 1e-3, on offline hidden states generated from ~4 500 UltraChat conversations (aux layers 2/24/45 + final, captured via the plugin's SupportsEagle3 interface and the extract_hidden_states connector).

Credit

Base model & architecture: IFM/K2-Horizon-MoVA-36B-A4B (MBZUAI Institute of Foundation Models). Apache-2.0.