DeepSeek-V4-Flash-MTP-GGUF

DeepSeek-V4-Flash MTP GGUF — the only GGUFs with the draft head intact

2026-08-13 — v2 files: MTP now works on STOCK llama.cpp. Upstream merged DeepSeek-MTP support (#25784). The Q8-MTP-v2/ and Q3_K_M-MTP-v2/ folders carry the same weights re-baked to mainline's metadata conventions — they run on any stock llama.cpp at or after that merge with --spec-type draft-mtp, no fork needed. Both smoke-verified here on stock mainline (Q8-v2 48.6 tok/s, Q3-v2 57.1 tok/s, single stream on our box). The original root-level v1 files remain for users of the July fork; v1 and v2 differ only in GGUF metadata (mainline counts the MTP layer in block_count = 44, per-layer arrays span 44, and the per-layer hc_head copies are dropped in favor of the global tensors) — the model bits are identical.

Every public DeepSeek-V4-Flash GGUF strips the model's multi-token-prediction (MTP) head: llama.cpp's converter drops the mtp.* tensors and its C++ has no code to run them. These are, as far as we know, the first and only V4-Flash GGUFs anywhere with the head preserved and working — the model drafts a token ahead of itself and verifies it in the same forward pass, for free decode speedup with zero quality change (every draft is verified by the full model).

Two files:

For reference, measured identically on the same hardware: the popular community Q4_K_M-XL scores 85.0 MMLU at 16.5 tok/s from a 175 GB file, and Unsloth UD-IQ3_XXS scores 79.6 at ~26 tok/s from 102 GB.

¹ MMLU = 120-question stratified sample; tool hardmode = tool-eval-bench adversarial suite (15 scenarios; we publish the range over 3 runs because the suite has ±10 pts of run-to-run variance — single-run scores you see elsewhere carry the same error bar, unstated). ² Single stream, MTP n_max=1, on 4× RTX PRO 4500 Blackwell (128 GB VRAM) + Threadripper 9970X, experts partially spilled to DDR5. Fully GPU-resident setups should see substantially more MTP gain (we measured 2.2× on a zero-spill control).

Why the "Q8" master beats every Q4 quant

DeepSeek released V4-Flash with the routed experts already in MXFP4 (4-bit). Our converter passes those bits through untouched and stores everything else (attention, sparse-attention indexer, MTP head, norms) at Q8_0. A "Q4_K" quant of this model is a sideways re-encode: it snaps weights that are already 4-bit onto a different 4-bit grid — a lossy round-trip that also produces a bigger file (Q4_K spends 4.5 bpw vs MXFP4's 4.25). Measured cost of that re-encode: 85.0 vs 88.3 MMLU, +15 GB.

Quantize downward to shrink (that's the Q3_K_M speed build). Never sideways.

Requirements

v2 files (recommended): stock llama.cpp at or after the DeepSeek-MTP merge (#25784) — build normally, serve with --spec-type draft-mtp --spec-draft-n-max 1.

v1 files (root of the repo): predate upstream support and need our July fork:

git clone --branch dsv4-mtp https://github.com/rogerai-fyi/llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build -j --target llama-server

The branch contains the converter patch, the MTP graph implementation for the deepseek4 architecture, and three unrelated bug fixes it flushed out (SIGFPE on empty KV-cache filters, quantizer crash on integer routing tensors, a metadata-ordering trap). A rebased-on-master variant lives on the same fork as dsv4-mtp-upstream.

Usage

# Q8-MTP master, 4×32 GB VRAM + DDR5 spill, MTP drafting on:
llama-server --model DeepSeek-V4-Flash-Q8-MTP-00001-of-00004.gguf \
  --n-gpu-layers 999 --n-cpu-moe 14 \
  --split-mode layer --tensor-split 20,8,8,8 \
  --no-repack --ubatch-size 128 \
  --flash-attn on --no-kv-unified --jinja --reasoning-format deepseek \
  --temp 1.0 --top-p 1.0 --min-p 0.0 \
  --spec-type draft-mtp --spec-draft-n-max 1

Notes from production (we serve the master daily on the box above):

• --spec-draft-n-max 1 is the sweet spot when experts spill to system RAM. Verifying N drafted tokens costs ~N× the DDR5 expert reads, so long drafts lose; a 1-token draft is accepted ~100% of the time (it is literally the head's training objective). The less you spill, the higher you can push it.
• Use mmap (default) — --no-mmap needs host RAM larger than the file.
• --no-repack is required at high --n-cpu-moe counts on current builds.
• Adjust --n-cpu-moe / --tensor-split to your VRAM; the Q3_K_M build needs --n-cpu-moe 10 --tensor-split 17,9,9,9 on the same hardware.
• KV cache stays f16: q8_0 KV corrupts V4 output (llama.cpp #24162). V4's sparse attention keeps KV small anyway.

Provenance

Converted directly from deepseek-ai/DeepSeek-V4-Flash (FP8/MXFP4 release) with our patched converter: mtp.* tensors rekeyed as layer 43, glue projections folded via the e_proj(e) + h_proj(x) = eh_proj(concat[e;x]) identity, all seven glue tensors verified against DeepSeek's reference implementation. The MTP head is a full extra transformer layer (MLA attention, 256-expert MoE, hyper-connection mixers — 1,575 source tensors). Metadata (block_count=44, nextn_predict_layers=1) is baked in.

Full build story, benchmarks and the speculation-economics analysis: https://rogerai.fyi/broadcasts-deepseek-mtp-gguf.html

DeepSeek-V4-Flash is MIT-licensed; these derivatives are too.

((•)) RogerAI — own your GPU. curl -fsSL https://rogerai.fyi/install.sh | sh