Kimi-K3-REAP-224experts

Kimi-K3-REAP-224experts

A 75% expert-pruned, English-only, quality-unvalidated research artifact derived from moonshotai/Kimi-K3.

This is not Kimi K3 and it does not perform like Kimi K3. None of the base model's published benchmark results carry over — assume they do not hold here, because 75% of the routed experts have been deleted with no recovery training. If you want a working K3, use the original weights.

🛑 We do not recommend using this checkpoint

Since publishing it we have built and used a 50% tier, Kimi-K3-REAP-448experts, and driven both with our own internal coding agent. The comparison is not close: the 448 build behaves like a usable model in hands-on use, and this one does not. Our own use of this checkpoint has stopped.

This is a subjective judgement from hands-on use, not a benchmark result — we have never scored either checkpoint. But it is a firmer statement than the original card made, and it points one way. If you want a working pruned K3, use the 448 build (or 384 if you only have 6 GPUs, though we have not tested that one).

This repository stays up because it is a useful artifact for research on where MoE pruning breaks — see Why publish it at all. It is not a model to deploy.

⚠️ Known to be broken in specific, reproducible ways

1. Non-English output is severely degraded — often unusable

The pruning decision was driven by a calibration corpus that is over 99% English (English code instructions, reasoning traces, and agentic coding traces; the only non-English content was incidental, at ~1.6% CJK characters in one component).

The REAP criterion scores an expert by its mean router-weighted activation over the tokens it actually sees during calibration. Experts specialising in languages the corpus never exercised score near zero and are therefore pruned first. Their capability is not degraded gracefully — it is removed.

Observed in practice: asked for a short story in Chinese, the model emits a heading and then locks into a single-token loop (the same character repeated ~2400 times) until the stream is cut. In the same forward pass, its English internal reasoning stays perfectly coherent. Chinese words also come back corrupted into non-words in otherwise-working output, and proper nouns garble (the model misnames its own lineage).

Treat this as an English-and-code-only model. Other languages are not "worse", they are broken. This limitation is a property of our calibration data, not of Kimi K3.

This appears to be fixable, and not by changing the ratio. Our 50% build was selected with Chinese added to the calibration corpus, and on that checkpoint the same prompts return fluent Chinese with no degenerate repetition. Note that build changed both the ratio and the corpus, so it is not a controlled result — but our per-expert measurements say the ratio alone would not have fixed it, because the affected experts sit at the bottom of the ranking at every cut point. The corpus is the part that matters.

2. Quality is not benchmarked, and degradation is expected even in English

No benchmark has been run on this checkpoint. What we know:

• Pruning to 224 experts discards 61.8% of the measured saliency mass. There is no knee in the curve here — marginal cost per pruned expert rises monotonically.
• We are not aware of any published result where a MoE survives >50% expert pruning without a healing phase. This model has none.
• In English coding tasks it produces working code but also writes semantically dead statements (e.g. a no-op x if False else None "placeholder"), and can corrupt a file further while trying to repair it, burning many turns in a repair spiral.

3. Multimodal path is untouched but untested

Only text MoE experts were pruned; the vision tower is intact. We have never run an image through it. Assume nothing.

Why publish it at all

The infrastructure result is reproducible and may be useful to others: a 3T-class MoE reduced from 1.56 TB to 475 GB by pure tensor surgery on packed MXFP4 weights, small enough to serve on a single 4×H200 node. It is also a concrete data point on what breaks first when you prune far past the safe range — the language-specific collapse above is, as far as we know, not documented anywhere in the expert-pruning literature.

If you are researching MoE sparsity limits, calibration-domain effects, or the interaction of low-bit quantization with pruning, this artifact is useful because it sits past the breaking point. If you want a model to use, it is not for you.

What was and wasn't touched

Each of the 92 MoE layers keeps the 224 highest-saliency routed experts out of 896. Because K3 ships in native MXFP4 (mxfp4-pack-quantized, group_size=32), removing an expert is pure tensor surgery on the packed weights — nothing is dequantized or requantized, and the per-group scales of surviving experts stay bit-for-bit valid, since only whole experts are ever dropped.

Kept experts stay in ascending original order, so router rows and expert indices remain consistent.

How the saliency was computed

REAP (router-weighted expert activation, mean-pooled over routed tokens). Scores are accumulated in the latent space (ℓ=3584): K3 applies routed_expert_down_proj before the per-expert loop, and the shared up-projection is identical across experts in a layer, so it cannot change their relative ranking.

Retained saliency mass by tier, from the same scores: 25% pruned → 83.6%, 50% → 63.3%, 75% (this model) → 38.2%.

Serving

Requires vLLM 0.27+ and trust_remote_code. At the time of writing the only working x86_64 path is the official image — the nightly wheels are aarch64-only.

docker run --gpus all -p 8000:8000 vllm/vllm-openai:kimi-k3-x86_64 \
  --model runrunway/Kimi-K3-REAP-224experts \
  --served-model-name k3-reap-224 \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --trust-remote-code \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser kimi_k3 \
  --reasoning-parser kimi_k3

Notes from bringing this up on 4×H200:

• --enable-prefix-caching must be passed explicitly; it is off by default for Kimi K3.
• Expert parallelism is preferable to pure tensor parallelism here — it keeps each MXFP4 expert whole on one rank instead of sharding quantized matrices across ranks.
• On Hopper the MXFP4 MoE dispatches to the Marlin W4A16 path, so weights stay 4-bit in memory. 475 GB of weights fit 4×H200 (564 GB) with room for KV cache; K3's 69 KDA layers carry constant-size recurrent state, so KV is far cheaper than a dense-KV transformer.
• If startup dies in flashinfer_autotune or cudagraph capture with cp_world_size must be positive, pass --no-enable-flashinfer-autotune and --attention-config '{"backend": "FLASHMLA"}'.

Tool calling did work reliably in our testing: structured tool-call arguments came back as valid JSON. That is a property of K3's typed tool-call format, not evidence that the model is otherwise healthy.

Other pruning ratios

Saliency is decoupled from the ratio — the same scores cut any tier without recomputation. If you want a different ratio, go shallower, not deeper: 672 experts (25% pruned) discards only 16.4% of saliency mass. Note that no ratio fixes the language problem, since the affected experts sit at the bottom of the ranking at every cut point; that needs a calibration corpus covering the target language and a fresh saliency pass.

We have since published two shallower tiers, both selected with a Chinese-inclusive corpus:

License and attribution

Derived from Kimi K3, © 2026 Moonshot AI, under the Kimi K3 License (included as LICENSE), which permits derivative works and their distribution provided the copyright and permission notice are retained. The base model, tokenizer, chat template, modeling code and all of the capability that survives here are Moonshot AI's work; this repository contributes only the expert selection and the surgery that applies it, and the defects described above are ours, not theirs.

Pruning criterion: REAP, from Cerebras' work on router-weighted expert activation pruning.

Provided as is, without warranty of any kind — see Section 5 of the license.