Kimi-K3-Nota-Global-Pruned-50

Nota Global-Pruned Kimi K3 (50%)

Highlight

Nota's global expert pruning is applied: saliency is measured on a global scale so that each layer keeps only the experts that matter to it. The result is a non-uniform checkpoint — every layer holds a different number of experts, 350 to 717 across the 92 MoE layers, 41,216 in total.

• The per-layer counts ship in config.json as n_routed_experts_per_layer, and the vLLM modeling file is patched to read it. Stock vLLM sizes every layer from the single scalar num_experts and cannot serve a non-uniform checkpoint.
• 50% pruning drops the deployment from B300 × 8 to B300 × 4.

Performance

Better than REAP (ICLR 2026), the method behind the pruned Kimi K3 variants published on Hugging Face.

HumanEval+ is the one benchmark where REAP scores higher, by 0.61pp. That is a single problem out of 164, and repeated runs of the same checkpoint moved by up to 1.22pp on this benchmark, so the gap is not a meaningful difference.

Evaluation config, per task:

• IFEval — thinking on, max_tokens=32768.
• GPQA-Diamond — thinking on, max_tokens=32768.
• IFBench — thinking on, max_tokens=32768.
• HumanEval+ — thinking on, max_tokens=32768.

More benchmarks will be added to this table.

Installation

Built against the vLLM version from the official Kimi K3 recipe, installed as documented there (as of 2026-07-27):

vllm 0.1.dev19262+gb6bbf29dd.d20260727

Replace model.py in that install with patch/model.py:

cp /workspace/kimi-k3/patch/model.py \
   /path/to/vllm/models/kimi_k3/nvidia/model.py

Find the exact path with:

python3 -c "import vllm, pathlib; print(pathlib.Path(vllm.__file__).parent / 'models/kimi_k3/nvidia/model.py')"

model.py.orig is the stock file, kept for restoring.

Quick Start

B300 × 4

VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1 \
FLASHINFER_PRIVATE_CUBIN_DIR=/opt/flashinfer-local-cubins/20260617_v0613rc1_situ_v0611_barrierfix \
CUDA_VISIBLE_DEVICES=0,1,2,3 \
vllm serve /workspace/kimi-k3/models/global_f350_50_kimi-k3 \
  --trust-remote-code \
  --load-format fastsafetensors \
  --moe-backend auto \
  --gpu-memory-utilization 0.95 \
  --tensor-parallel-size 4 \
  --max-num-seqs 256 \
  --no-enable-flashinfer-autotune \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --reasoning-parser kimi_k3

--max-num-seqs 256 matters: KDA state is pre-allocated per sequence out of the same pool as the attention KV cache, and vLLM's default of 1024 aborts at CUDA-graph capture with max_num_seqs (1024) exceeds available Mamba cache blocks (962).

Do not add --enable-expert-parallel — EP requires num_experts % ep_size == 0, which a non-uniform checkpoint cannot satisfy.

See /workspace/kimi-k3/inference/run_global_f350_50.sh for the same launch with a pre-flight memory check.