kimi-k2.5-eagle3-mla

kimi-k2.5-eagle3-mla

Model Overview

kimi-k2.5-eagle3-mla is an Eagle3 MTP draft model with MLA(Multi-Latent-Attention) for accelerating inference of Kimi-K2.5, trained with TorchSpec - an online speculative decoding training framework that runs FSDP training and inference concurrently. If you find this draft model useful, please give our project TorchSpec a star on GitHub.

Why an MLA (Multi-Latent Attention) Draft Model

Compared with an MHA draft model, the MLA variant is a better fit for Kimi-K2.5 deployment:

• Uses less KV cache, which reduces serving memory pressure.
• Matches Kimi-K2.5's MLA architecture, so it fits more naturally into the inference engine's KV-cache handling under different serving scenarios such as PD-Disaggregation.

Training Setup

• Cluster: 4 nodes x 8x H200 (32 GPUs total)
• Training: 2 nodes (16 GPUs), FSDP
• Inference: 2 nodes (16 GPUs), Engine (TP=8 per node)
• Duration: ~14 hours per phase:

Dataset: Regenerated open-perfectblend dataset

All training responses were regenerated by Kimi-K2.5 via Engine to match the base model's exact token distribution.

Performance

The primary metric is accept_length - the average number of tokens accepted per speculation step with topk=1, num_steps=3, num_draft_tokens=4. Higher is better.

Benchmarks were run using lm_eval.

---

Quick Start

Requirements

• NVIDIA GPU with CUDA 12.0+
• vLLM >= 0.18.0, or install the nightly wheel/docker image
• The model is supported in SGLang latest main. Refer to the official SGLang installation guide.

Launch Server (vLLM)

vllm serve moonshotai/Kimi-K2.5 \
    --tensor-parallel-size 8 \
    --speculative-config '{"model": "lightseekorg/kimi-k2.5-eagle3-mla", "method": "eagle3", "num_speculative_tokens": 3}' \
    --trust-remote-code

For deployment configuration, refer to the official vLLM recipes.

Launch Server (SGLang)

sglang serve \
  --model-path moonshotai/Kimi-K2.5 \
  --tp 8 \
  --trust-remote-code \
  --reasoning-parser kimi_k2 \
  --tool-call-parser kimi_k2 \
  --speculative-algorithm EAGLE3 \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --speculative-draft-model-path lightseekorg/kimi-k2.5-eagle3-mla

For deployment configuration, refer to the official SGLang cookbook.

Run Benchmarks

lm_eval \
  --model local-completions \
  --model_args base_url=<url> \
  --tasks gsm8k \
  --batch_size 16