K3-Stuff

K3-Stuff

Artifacts + tooling for running Kimi K3 (2.78T MoE, 896 routed experts, MXFP4 QAT) with DSpark/DFlash speculative decoding in llama.cpp, validated on rented 8× RTX 5060 Ti hardware before committing to a 2× RTX 3090 home build.

Draft model (root)

• K3-DSpark-draft-Q8_0.gguf — K3 DSpark/DFlash MTP draft, Q8_0, rewritten for llama.cpp:
  • arch renamed dflash-draft → dflash, tensor names fixed, K3 tokenizer keys added
  • tokenizer.ggml.mask_token_id=163824 added (from dflash.mask_token_id) — required by llama.cpp speculative.cpp
• fix_draft_gguf.py — original GGUF rewrite (KV/tensor renames, tokenizer keys from K3)
• add_mask_token.py — adds mask_token_id to the rewritten draft

llama-cpp-patches/ — modified llama.cpp (apply with git apply or patch -p1)

Three changes, base commit ~b10630 (d222767c7):

1. llama-arch.cpp — accept dflash-draft as an alias for the dflash arch.
2. llama-model.cpp — LLAMA_MMAP_NO_PREFETCH=1 env disables MAP_POPULATE/prefetch at load. Without it llama.cpp faults in the whole 1.4TB model and OOMs when RAM < model size. With it, the page cache becomes the hot-expert cache (lazy fault-in + kernel LRU eviction). Required for any sub-full-RAM K3 box.
3. kimi-k3.cpp — populate res->t_layer_inp[] for the layers DSpark taps (7/23/51/67/83 via cparams.embeddings_layer_inp). Without this, DSpark crashes at llama-graph.cpp: GGML_ASSERT(t_layer_inp[il] != nullptr). Mirrors deepseek4/bailingmoe3. Upstream PR candidate.

scripts/ — full experiment + tooling suite

Highlights:

• requant_trunk.c — surgical GGUF rewriter: requants trunk Q8_0→Q4_K while byte-preserving MXFP4 experts + F32 norms (llama-quantize can't do this safely — it requants the experts and destroys QAT calibration). K3's trunk is Q8_0 (59.6GB); Q4_K trunk = 34.2GB GPU-resident → fits 2×3090 with room for a GPU-resident DSpark draft. Build: gcc -O2 -c requant_trunk.c -I<llama.cpp>/ggml/include -I<llama.cpp>/ggml/src then link with g++ ... libggml-base.a libggml-cpu.a libggml.a -lm -lpthread -fopenmp.
• 30_requant_all.sh — driver: requant all 32 shards in parallel with write→verify→delete-source (for disk-constrained boxes).
• measure_quants.py — walk all shards, report byte/tensor split by quant category (this is how we discovered the trunk was Q8_0 not 4-bit).
• 28_kitchen_sink.sh — main perf experiment (FA on, cpu-moe, DSpark spec, tensor-split).
• 20/21_* — baseline perf suites; 25_expert_residency.sh, 26_draft_sweep.sh — staged follow-ups.
• FINDINGS.md / HANDOFF.md — full root-cause log + session handoff (the durable record).

Key facts

• K3 is QAT-trained in MXFP4 — the 4-bit experts ARE the reference model. Never re-quant them.
• Usage: LLAMA_MMAP_NO_PREFETCH=1 llama-server -m <K3 shard1> -ngl 999 --tensor-split ... --cpu-moe -fa on -t 112 -md K3-DSpark-draft-Q8_0.gguf -ngld 999 --spec-type draft-dspark