KAT-Coder-V2.5-Dev-MTP-APEX-GGUF

KAT-Coder-V2.5-Dev — APEX GGUF with MTP head

MoE-aware, mixed-precision APEX quantization of Kwaipilot/KAT-Coder-V2.5-Dev, an agentic-coding fine-tune on the Qwen3.6-35B-A3B backbone (qwen3_5_moe: 40 layers, 256 routed

• 1 shared expert, hybrid GatedDeltaNet linear-attention with periodic full attention). Kwaipilot reports SWE-bench Verified 69.40 for this size class, against Qwen3.6-35B-A3B's 64.40.

What makes this build different: it has a working MTP head. KAT-Coder ships mtp_num_hidden_layers: 0 — no multi-token-prediction head at all, so no speculative decoding is possible out of the box, and that is true of the vendor release and of every other quant of this model I am aware of. This repo transplants Qwen3.6-35B-A3B's own trained MTP head onto KAT-Coder and quantizes the result. Measured below: 2.03x on a hard agentic-coding suite, with correctness unchanged.

Recommended speculative settings

llama-server -m KAT-Coder-V2.5-Dev-MTP-APEX-i-quality-v2.gguf \
    --spec-type draft-mtp --spec-draft-n-max 3

Full draft-depth x temperature surface, measured on -v2. One long-context prompt (~82 KB), max_tokens=400, 3 seeds per cell, median tok/s, top_p 0.95 top_k 20, ctx 32768, build 6d05498. Speedup is against the no-drafter baseline at the same temperature:

Draft acceptance for the same cells:

Depth 3 wins at every temperature, and the falloff past it is steep: acceptance decays roughly geometrically with depth while draft compute grows linearly, so at n-max 8 the wasted work exceeds the gain and it runs slower than no drafter at all (0.82-0.94x). If you tune one knob, tune this one.

The no-drafter row is flat across temperature (59.2-59.4), which is the control: all the variation above it is acceptance, not decode speed.

Absolute tok/s here is lower than the from-scratch figures further down (59.3 vs ~70.8 baseline) because this sweep uses a long-context prompt — decode slows as context grows. The ratios are the transferable part.

Every number in both tables is throughput. Correctness is unaffected by construction: the target model verifies each drafted token against its own computation, so a rejected draft never reaches the output. Speculative decoding cannot change what the model says, only how fast it says it.

Files

Pick MTP-APEX-i-quality unless you are VRAM-constrained, in which case APEX-dynamic is the best quality available at its size. To run either you need only the .gguf. The safetensors shard and index are there for anyone who wants to redo the splice against the original checkpoint: drop those three files into a local copy of Kwaipilot/KAT-Coder-V2.5-Dev alongside its own 13 model-*-of-00013.safetensors shards. KAT-Coder's original shards are untouched and byte-identical; this is a drop-in addition, not a re-upload of the 65 GB checkpoint.

The MTP head — honest framing

This is the naive, un-fine-tuned transplant. The head's weights are copied straight from stock Qwen3.6 with zero training on KAT-Coder's own hidden-state distribution. It is architecturally clean rather than a hack — hidden_size, expert count, and attention/MoE shapes match exactly, because KAT-Coder is itself built on the Qwen3.6-35B-A3B backbone — but a fine-tuning pass (freezing KAT-Coder's 40 layers, training only the ~850M-parameter head) is a planned follow-up. These numbers are the baseline before that.

Draft-acceptance rate

llama-server --spec-type draft-mtp --spec-draft-n-max 2, 5 real from-scratch coding prompts, n_predict=400 each:

The same quant tier without any MTP head runs ~70.8 tok/s solo. Acceptance tracks predictability: BST traversal boilerplate hits 89%, the LRU cache with genuine locking and eviction logic is hardest at 61%.

Agentic coding benchmarks

Two task suites run through opencode's real tool-calling loop, MTP build vs the same quant without MTP. Correctness was identical — 100% on both suites either way (59/59 and 28/28 test cases), which is what the theory predicts: the MTP head only drafts, the main model always verifies against its own computation, so a bad draft is rejected and never reaches the output. Only speed moves.

The single slower result (lru_ttl_cache, 0.81x) is most likely run-to-run variance in how much iteration that attempt needed rather than a systematic MTP cost — every other task improved.

v2

Two allocation defects were found and fixed. Both were reported or surfaced after the original upload; the -v2 files are the corrected builds and the originals are retained so earlier numbers stay reproducible.

1. State coefficients were quantised. blk.N.ssm_alpha and blk.N.ssm_beta — 60 tensors, [2048,32], 3.93 M params total — were absent from the allocation recipe and silently inherited the base quant type (Q4_K here, Q3_K in the dynamic tier). They now ship F32. These parameterise a gate: beta = sigmoid(ssm_beta . x), alpha = softplus(ssm_alpha . x + dt), and the result multiplies the state path, so error there compounds along the sequence rather than adding to one layer's output. Cost of pinning them: ~14 MB on a 19 GiB file. Reported by @anjeysapkovski.

2. The MTP block was never allocated. blk.40 — the transplanted head, 805 M params of experts plus nextn.eh_proj — was also missing from the recipe and took the base type by accident, landing four expert tensors on Q6_K. A drafter's precision buys acceptance rate, never output quality, so those were bits spent where the return is zero by construction. blk.40 is now pinned Q4_K throughout. Measured effect on acceptance: none (within run-to-run spread at every depth), and the file is 56 MiB smaller.

Both v2 files pass a recipe-vs-artifact audit: every requested tensor type honoured, and zero tensors quantised that the recipe never asked about (the original files had 71 such tensors).

The imatrix

llama-imatrix is impractical on this architecture: GatedDeltaNet linear-attention is a serial state-space recurrence, and the imatrix collection callback breaks the GPU path, falling back to a single CPU thread. kat-coder.imatrix was instead generated with a band-serialized PyTorch forward pass over a general text/code corpus. It is a standard GGUF-format importance matrix (in_sum2 + counts per tensor), bit-compatible with llama-quantize --imatrix, so it is directly reusable if you want to build your own tiers.

One checkpoint quirk worth knowing if you do: KAT-Coder stores MoE experts as fully separate per-expert tensors (mlp.experts.{e}.{gate,up,down}_proj.weight × 256) rather than the fused gate_up_proj/down_proj layout the HF module expects. Both imatrix generation and GGUF conversion have to fuse them.

Allocation

APEX structural mode: routed experts are bucketed into edge / near / middle depth bands (edge = first and last layers, generally most load-bearing) with decreasing precision toward the middle. For i-quality that is routed edge → Q6_K, near → Q4_K, middle → Q3_K. Attention, shared-expert, and router tensors stay at higher fixed precision (Q6_K–Q8_0).

Measured quality

Wikitext perplexity across the tiers that were built. The two marked hosted are in this repo; the other two were measured for comparison and are not uploaded:

Two things worth drawing out. i-quality lands within 1.3% of full bf16 at under a third of the size. And APEX-dynamic beats the stock IQ3_XXS baseline on both axes at once — smaller (12.13 vs 12.68 GiB) and lower perplexity (7.543 vs 7.881) — which is why it is the one hosted alongside the flagship rather than the stock tier.

That result is cleaner than the same method got on Laguna-XS, where it traded size for a small PPL cost. Whether role/depth-aware allocation beats a well-tuned stock quant appears to depend on the architecture and calibration, and is not something to assume without measuring.

The hosted MTP build uses the i-quality recipe; its file is slightly larger than the 19.23 GiB row because it carries the MTP head as well.

Attribution

• KAT-Coder base model: Kwaipilot — Kwaipilot/KAT-Coder-V2.5-Dev (Apache-2.0).
• MTP head weights: Qwen — Qwen/Qwen3.6-35B-A3B (Apache-2.0), transplanted unmodified.
• APEX recipe & toolkit: LocalAI — localai-org/apex-quant.
• Quantization engine: llama.cpp (ggml-org).

Unofficial community quantization and experiment; not affiliated with or endorsed by Kwaipilot or Qwen.