Kimi-K3-REAP80-MLX-mxfp4-q8

Kimi-K3-REAP80-MLX-mxfp4-q8

Kimi-K3, REAP expert-pruned (80%) and converted to MLX — 350 GB, sized to actually load on a 512 GB Apple Silicon machine.

Base: moonshotai/Kimi-K3 — 2.78T total / 104B active, native-multimodal MoE with Kimi Delta Attention, Attention Residuals and a 1M context.

Why this exists

Unpruned Kimi-K3 does not fit on any Mac. The full model is 1.56 TB; even a 2-bit quant is ~870 GB against a 512 GB ceiling. Fitting 4-bit into 512 GB would need ≤1.38 bits/weight.

REAP (Cerebras) scores each expert by gate x ||expert_output|| over a calibration set and keeps the most salient. This build keeps 179 of 896 experts per layer across 92 MoE layers → 601B params, 350 GB.

Precision: mxfp4, and it is lossless

K3's routed experts ship from Moonshot as MXFP4 (weight_packed + e8m0 weight_scale, group 32). MLX's native mxfp4 mode uses the identical encoding, so the surviving experts here are a bit-exact byte copy of the source — the only information lost in this repo is the pruning itself, not the quantization.

Requantizing those same weights to affine 4-bit would cost ~9.8% mean relative error and be larger (4.5 vs 4.25 bits/weight), so no affine 4-bit tier is published.

Non-expert tensors (mxfp4 global config; attention, shared experts, latent projections, embeddings) are carried at higher precision.

Measured behaviour

Loaded on a 512 GiB M3 Ultra (52 s, 349 GB peak). Verbatim, greedy, unedited:

prompt: 'def merge_intervals(intervals):\n    """Merge overlapping intervals."""\n'
--> intervals.sort(key=lambda x: x[0])
    merged = []
    for interval in intervals:
        if not

prompt: 'The capital of France is'
--> Paris.",
+    "The capital of Germany is Berlin.",
+    "The capital of Italy is Rome.",
+

prompt: '机器学习的基本原理是'
--> ，机器学习是一种基于人工智能的学科，是一种基于人工智能的学科，是一种基于人工智能的学科，是一种基于人工智能的

Samples above are verbatim, greedy-decoded, and unedited — including the failure modes. Degradation from the pruning shows up as drift into list-like or source-file-like continuations rather than answering directly, and at heavier prune ratios as outright repetition loops.

Chinese output degrades noticeably in this build — the sample above loops. Prefer the REAP-73 build if you need CJK.

Code completions stay structurally correct across every ratio tested, which matches the calibration data: code experts form a dense, self-similar cluster (57% self-overlap in a top-242 set, versus a 27% chance baseline) and so survive pruning better than more diffuse language capability.

Speed

~5.54 tok/s decoding on a 512 GiB M3 Ultra, via mlx_lm.generate or any standard mlx-lm entry point. Prompt processing is considerably faster; the figure above is generation.

Each decoded token reads roughly 87 GB of weights: 25.8 GB of routed experts plus 60.8 GB of non-expert tensors, which every token touches. Against the M3 Ultra's ~819 GB/s that puts the ceiling near 9.5 tok/s, so this build reaches about 59% of what the memory system permits. It is bandwidth-bound, which is the expected regime for a model of this shape.

Note the non-experts dominate per-token traffic despite being ~2% of parameters: all of them are read every token, while only 16 of 179 experts are.

Pruning buys memory, not speed. Per-token traffic depends on top_k and the non-expert precision, never on how many experts are stored, so the 350 GB 179-expert build and the 451 GB 242-expert build both decode at ~5.5 tok/s. Prune harder to fit a smaller machine, not to go faster.

Correction, 2026-07-28. Earlier revisions of this card reported ~0.20 tok/s and described this build as "not interactive". That number came from an internal benchmark that hand-rolled its own decode loop and so never entered the wired_limit context manager mlx_lm.stream_generate applies automatically. The weights were left unwired, and every decoded token faulted them back from SSD instead of reading RAM -- understating this build by roughly 28x. Users were never affected: the normal mlx-lm paths (generate, stream_generate, the CLI, the server) have always wired correctly, so what you measure is the corrected figure above. Thanks to @pudepiedj for reporting the discrepancy.

Quality expectations — read this

This is an aggressive prune. Top-16 routing over 179 experts is 8.9% density, comparable to a REAP-50 of a 256-expert model. Expect noticeable degradation versus full K3. It is the "fits on one machine" build, not a quality build.

Two things work in its favour that a plain expert cull would not have: K3 keeps 2 shared experts that fire on every token regardless of pruning, and its LatentMoE applies RMSNorm to the combined expert output, which partially self-corrects the magnitude lost when experts are removed.

Calibration

Saliency was measured on a deliberately mixed 12.6 MB corpus — 40% code (multi-language + real Python), 30% English web, 15% Chinese, 15% across ja/ru/ko/de/fr/es/ar. The mix matters: whatever a calibration corpus under-represents gets pruned away silently. An earlier attempt using C4's pooled multilingual config left CJK at 0.03% of the corpus, which would have quietly removed the experts handling Chinese.

Usage

Requires mlx-lm plus the bundled kimi_k3.py loader (the architecture is not upstream yet). On a 512 GB machine you must raise the GPU wired limit first — the default is ~75% of RAM, below this model's footprint:

sudo sysctl iogpu.wired_limit_mb=480000
pip install mlx-lm

python - <<'PY'
import os, shutil, mlx_lm
from huggingface_hub import hf_hub_download
for f in ("kimi_k3.py",):
    dst = os.path.join(os.path.dirname(mlx_lm.__file__), "models", f)
    shutil.copy(hf_hub_download("pipenetwork/Kimi-K3-REAP80-MLX-mxfp4-q8", f), dst)
PY

mlx_lm.generate --model pipenetwork/Kimi-K3-REAP80-MLX-mxfp4-q8 --max-tokens 256 \
    --prompt "Write a Python function that merges overlapping intervals."

kimi_k3_vision.py and kimi_k3_vl/ ship alongside for the vision tower; the image path needs mlx-vlm and is not exercised by mlx_lm.generate.

Provenance

Converted with PipeNetwork/kimi-k3-mlx: a streaming converter (the model never fits in memory at any stage) and a streaming REAP calibration harness. Weights remain under the Kimi K3 License.