DeepSeek-V4-Flash-MTP-Only-GGUF

DeepSeek-V4-Flash MTP head (GGUF)

This is the standalone multi-token-prediction (MTP/NextN) head from deepseek-ai/DeepSeek-V4-Flash, converted to GGUF for use as a speculative-decoding draft in ik_llama.cpp. It cannot be used by itself.

Why this exists

ik_llama.cpp supports DeepSeek-V4 MTP via a standalone companion GGUF (#2216). The DeepSeek-V4-Flash checkpoint contains a single NextN head with the tensor layout that the ik_llama.cpp converter expects. This is a conversion of that MTP head.

The DeepSeek-V4-Flash-0731 checkpoint elides this MTP head, but the original checkpoint's head/this conversion is still compatible with it. The only downside is that its acceptance rate is likely to be lower than that of a head for 0731 specifically. As the MTP head only proposes tokens that are then validated, there is no risk of misgeneration.

Usage

ik-llama-server \
  -m  <any DeepSeek-V4-Flash GGUF> \
  -md DeepSeek-V4-Flash-MTP-bf16.gguf \
  --spec-type mtp:n_max=1,p_min=0.0 --spec-ckpt-mode per-step \
  -fa on ...

Use n_max=1. Deeper MTP drafts lose on measurement: at n_max=3, acceptance drops from ~0.74 to ~0.42 and generation lands at or below the no-draft baseline (this may be an artifact of the prediction being weaker for 0731).

The file also loads on recent mainline llama.cpp via --spec-type draft-mtp. That compatibility is untested; mainline users are likely better served by the mainline-supported DSpark draft that 0731 ships.

Measurements

Target: DeepSeek-V4-Flash-0731 IQ2_XXS-slim, 128k context, 2×RTX 3090 + Threadripper 3960X with hybrid CPU offload, greedy decoding.

Acceptance is flat with prompt depth. Acceptance transfers across hardware, but the throughput column doesn't (it is dependent on verification cost).

Provenance

Converted with ik_llama.cpp's convert_hf_to_gguf.py --mtp --outtype bf16 (rev c013cd8) from deepseek-ai/DeepSeek-V4-Flash shards model-00001-of-00046, model-00045-of-00046, model-00046-of-00046.

The conversion needs torch ≥ 2.7 for float8_e8m0fnu; the repo's pinned torch 2.2 fails.

bf16 describes the attention/norm/embedding tensors. The MoE experts remain MXFP4 (~4.25 bpw) as shipped by DeepSeek; there is no further room to quantize them.

sha256: 3efae2854ca205a338f536a434649c1007df22ccfd72b102aa006c3bb5bc912c

ycui7/DeepSeek-V4-Flash-MTP published the same head as safetensors first. That repo omits the root tensors, which means ik's converter cannot naturally make use of them.

License

MIT, inherited from the source model.