Kimi-K3-GGUF

Kimi-K3 UD-IQ1_S GGUF

Kimi-K3 (2.78 T parameters) compressed to 330.2 GB IQ1_S GGUF (307.49 GiB, 9 shards) — about 21% of the ~1.56 TB source artifact.

Second pack: UD-IQ1_S_DENSE/ is a separate quant, 320.33 GB (9 shards), the dense-path variant — not a second IQ1_S. Do not mix the two shard sets.

Every one of the 82,432 routed experts is present — no experts are dropped. The GGUF loads without model-specific patches on an existing Kimi-K3-capable llama.cpp build.

The exact method used to create this model will be released soon. This card documents what the model is, how to run it, and how it measures — not how it was built.

Benchmarks

---

1. SparkInfer TP3/TP4 — Optimal Path (Blackwell)

Current measured throughput: 12.5468 and 12.5516 tok/s, averaging 12.5492 tok/s around a matched 6.4930 baseline, with 112/112 accepted drafts and identical token IDs. This is the K=8 repeat-heavy structured profile from the SparkInfer TP3 recipe, verified on real DGX Spark (GB10) hardware.

SparkInfer is optimized for all Blackwell chips (H200, GB10/DGX Spark, and other GB10-based systems). The gittensor-ai-lab/sparkinfer-k3 repo runs on 8× H200 as the reference configuration; DGX Spark (GB10) has a reduced footprint (~113 GiB/rank under 121 GiB).

Why MY SparkInfer Patch Is Optimal

• Correctness verified 2026-08-11: The distributed (TP3/TP4) generation path had a bug in kimi_k3_dist_rank.cpp: it never set the GGUF capability flags that gate loading, so shared experts, routed-expert normalization, and MLA attention gate were silently skipped on every token — real weights present, never computed. MY fix adds the same capability probe every other code path already had.
• Footprint: ~113 GiB/rank fits comfortably under GB10's 121 GiB.
• Current measurements (corrected engine):
  • Load time: ~5m45s/rank (was ~30-60 min pre-patch 0013)
  • SPARKINFER_K3_KDA_FUSE=0: +20% over MY own baseline
  • K=4 speculative decoding candidate: 10.3002 and 11.3231 tok/s (10.8117 avg), +68.2% over matched 6.4272 baseline, 95/95 accepted drafts
  • K=8 repeat-heavy profile: 12.5468 and 12.5516 tok/s (12.5492 avg) around 6.4930 baseline, 112/112 accepted drafts, identical token IDs
• Not valid for non-Blackwell chips: Tested on physical fleets, hard memory ceiling on other hardware.

Full launch procedure (see kimi-k3-neuron-tp3-dgxspark-recipe):

1. 3 or 4 DGX Spark (GB10) nodes, each with ≥320 GB free NVMe
2. Pull model with hf_xet transport
3. Apply verified 28-patch chain through 0026
4. Build SparkInfer with MY patch series
5. Copy built dist/ tree to every rank node
6. Launch rank 0 (coordinator) with sparkinfer --model ./Kimi-K3-UD-IQ1_S/UD-IQ1_S --tp 3 --rank 0 --listen --ctx-len 8192 --jdba-band
7. Launch rank 1 and rank 2 (workers) with --coord <rank0-host>:9000 --jdba-band
8. Verify all ranks connected, then query

Shards load from UD-IQ1_S/Kimi-K3-UD-IQ1_S-00001-of-00009.gguf (all 9 parts required).

Speculative Decoding (Experimental, Opt-In)

Patches 0021–0026 add n-gram/prompt-lookup speculative decoding, match-confidence gating, recursive-majority continuation, and distributed LM-head banding. On real TP3 hardware, 64-token generations show:

• Code/structured: +27% with --spec-draft 4
• Literal repetition: +83% with --spec-draft 4
• Freeform prose: −3.7%
• Mean across all three: +34.5%

---

2. llama.cpp — Fallback (Any Chip)

Requires a llama.cpp build with Kimi-K3 support — upstream PR #26185 or the Unsloth K3 fork.

llama-server \
  -m UD-IQ1_S/Kimi-K3-UD-IQ1_S-00001-of-00009.gguf \
  -ngl 99 --jinja \
  -c 8192 --parallel 1 \
  -ctk f16 -ctv f16 -fa on \
  --temp 1.0 --top-p 0.95

---

3. vLLM TP3 + DSpark — Last Resort (Non-Blackwell)

---

Download

This repository is Xet-backed. Use the hf_xet transport — chunk-deduplicated and parallel:

pip install -U "huggingface_hub[hf_xet]"
hf auth login

export HF_XET_HIGH_PERFORMANCE=1
hf download vcruz305/Kimi-K3-GGUF --local-dir ./Kimi-K3-UD-IQ1_S \
   --include "UD-IQ1_S/Kimi-K3-UD-IQ1_S-*.gguf" --include "k3_chat_template.jinja"

All 9 shards are required; llama.cpp loads them automatically from UD-IQ1_S/Kimi-K3-UD-IQ1_S-00001-of-00009.gguf.

Dense-path variant only (320.33 GB, not a second IQ1_S):

hf download vcruz305/Kimi-K3-GGUF --local-dir ./Kimi-K3-UD-IQ1_S_DENSE \
   --include "UD-IQ1_S_DENSE/Kimi-K3-UD-IQ1_S_DENSE-*.gguf" --include "k3_chat_template.jinja"

Load that pack from UD-IQ1_S_DENSE/Kimi-K3-UD-IQ1_S_DENSE-00001-of-00009.gguf.

Two things learned moving this artifact around:

• Xet dedup is per chunk, not per file — when shard 00001 was rewritten to embed the chat template, the 39.4 GB re-upload transferred 395 kB.
• Advertised bandwidth is meaningless: a host advertising 538 Mbps sustained 7,828 Mbps on real Xet transfers.

---

Status

Experimental, and it works. It loads, reasons in K3's native format, and scores 94.5% on HumanEval — measured on the full 164 problems, not a subset.

---

Changelog

• 2026-09-26: Split the two packs so the Hub lists them separately. Reference shards moved to UD-IQ1_S/. The dense-path variant is UD-IQ1_S_DENSE/ (320.33 GB), previously Q6-dense/Kimi-K3-UD-IQ1_S-Q6D-*. Server-side pointer move; no weight re-upload. Old root and Q6-dense/ URLs 404.
• 2026-08-12: Renamed GGUF shards to Kimi-K3-UD-IQ1_S-*-of-00009.gguf (server-side pointer rename; no weight re-upload).
• 2026-08-11: Correctness fix applied and verified in SparkInfer TP3 engine — kimi_k3_dist_rank.cpp now sets GGUF capability flags that gate loading of shared experts, routed-expert normalization, and MLA attention gate. Pre-fix measurements were invalid (engine skipped real computation).
• Pre-2026-08-11: Historical speed tables deprecated — measured on engine skipping computation.
• Initial release: 330 GB IQ1_S GGUF with all 82,432 experts present, verified loading on Unsloth llama.cpp fork.