DeepSeek-V4-Pro-0813-GGUF

DeepSeek-V4-Pro-0813 GGUF

Sub-4-bit GGUF quantizations of deepseek-ai/DeepSeek-V4-Pro-0813, produced by the 6block team with importance-matrix (imatrix) calibration.

1.57T parameters, 48B active per token. 61 layers, 384 routed experts (top-6) + 1 shared expert.

Why only sub-4-bit tiers

The upstream weights ship in FP4 (expert_dtype: fp4 in config.json). Routed-expert tensors are stored pre-packed, so convert_hf_to_gguf.py writes them straight to GGUF's MXFP4 type without ever materialising BF16. The resulting F16 GGUF is 812.7 GiB at 4.33 bpw, and expert tensors are 96.4% of it.

That means the usual "higher tier = better" ladder does not apply. Measured with llama-quantize --dry-run against this exact master:

Quantizing a 4.25-bpw tensor up to 8 bpw only doubles the file; it cannot recover information the factory FP4 step already discarded. Everything published here is below the master's 4.33 bpw.

For 4-bit and 8-bit builds of this model, see unsloth/DeepSeek-V4-Pro-0813-GGUF (UD-Q4_K_XL 850 GB, UD-Q8_K_XL 873 GB). This repository covers the range below that.

Available quantizations

Sharded tiers

The top three tiers exceed HuggingFace's 500 GB per-file limit, so they ship as …-NNNNN-of-NNNNN.gguf shards of roughly 42 GiB each. Download every shard of a tier into one directory and point -m at the first one — llama.cpp reads split.count from shard 00001 and pulls in the rest automatically. Do not try to concatenate them; they are individually valid GGUF files, not split-style byte chunks.

Master F16 GGUF baseline: PPL 4.0795 ± 0.0458 (measured during imatrix, 220 chunks — a different chunk count than the table above, so it is not directly comparable; see caveats).

The quality cliff sits between Q2_K and IQ1_M: 200 GiB of savings costs +1.93 PPL, whereas the entire Q3_K_M → Q2_K range costs only +0.14.

Tiers deliberately not published

IQ2_XS (445 GiB, PPL 4.4705) and IQ2_XXS (401 GiB, PPL 21.6798) were built and then rejected. Both are beaten outright by smaller files — IQ1_S is 315 GiB at PPL 4.11 — so they occupy a size bracket while delivering worse output. The IQ2 expert-quantization path appears to break down on this sparse-routing MoE; the same failure mode showed up on DeepSeek-V4-Flash's IQ2_M. Sizes and tensor counts looked completely normal, which is why every tier here was PPL-tested before release.

Quantization details

• Tool: llama.cpp @ 4ed2b13 (needs LLM_ARCH_DEEPSEEK4; older builds reject deepseek4)

• imatrix: 220 chunks over a 476 KB multilingual corpus (EN/ZH), final PPL 4.0795, published as imatrix.gguf

• Requantization: --allow-requantize is mandatory. Expert tensors arrive already quantized as MXFP4, and llama.cpp refuses to requantize by default (requantizing from type mxfp4 is disabled). Note this makes every tier here a second quantization pass on top of the factory FP4 step.

• Non-expert tensors are protected explicitly, because a global low-bit setting would otherwise crush the sparse-attention indexer and the per-layer control tensors:

  hc_*                    → F32     (per-layer control)
  attn_q/k/v/output       → Q8_0
  indexer*, compressor*   → Q8_0    (sparse-attention index path)
  ffn_gate_inp            → F32     (router)
  shexp                   → Q8_0    (shared expert)
  token_embd, output      → Q6_K
  

  --tensor-type matches substrings and first match wins, so attn_ alone would also swallow hc_attn_fn. The four attention projections are listed separately on purpose.

• Metadata: general.quantized_by=6block, no absolute paths in any KV field.

Usage

llama.cpp

Sharded tier (Q3_K_M / IQ3_XXS / Q2_K) — fetch all shards, then load the first:

hf download 6block/DeepSeek-V4-Pro-0813-GGUF \
  --include "DeepSeek-V4-Pro-0813-IQ3_XXS-*.gguf" --local-dir .

llama-server -m DeepSeek-V4-Pro-0813-IQ3_XXS-00001-of-00015.gguf -c 8192 --jinja

Single-file tier (IQ1_M / IQ1_S):

hf download 6block/DeepSeek-V4-Pro-0813-GGUF \
  DeepSeek-V4-Pro-0813-IQ1_S.gguf --local-dir .

llama-server -m DeepSeek-V4-Pro-0813-IQ1_S.gguf -c 8192 --jinja

Do not pass -ngl or --n-cpu-moe manually. Setting either makes llama.cpp abandon automatic VRAM fitting and split by layer count instead, which overflows individual cards on a model this size (common_fit_params: n_gpu_layers already set by user to 99, abort, then cudaMalloc failed). Let it fit the model itself.

Ollama

FROM takes one file, so for a sharded tier merge the shards first (needs free space for both the shards and the merged result):

llama-gguf-split --merge \
  DeepSeek-V4-Pro-0813-IQ3_XXS-00001-of-00015.gguf \
  DeepSeek-V4-Pro-0813-IQ3_XXS.gguf

cat > Modelfile <<'EOF'
FROM ./DeepSeek-V4-Pro-0813-IQ3_XXS.gguf
PARAMETER temperature 0.6
PARAMETER top_p 0.95
EOF

ollama create deepseek-v4-pro -f Modelfile
ollama run deepseek-v4-pro

Caveats

Read these before comparing numbers with any other repository.

1. PPL is wikitext-2, n_ctx=512, 12 chunks. Cross-tier comparisons in the table are valid; comparisons against other models or other repos' published figures are not. Perplexity's running average climbs monotonically as more corpus is covered, so a 12-chunk number and a 568-chunk number are different measurements even for the same file.
2. The 4.0795 master baseline was measured at 220 chunks, during the imatrix pass — not at 12. It indicates the master's general range, not a like-for-like delta against the table.
3. Every tier is a double quantization (factory FP4 → MXFP4 → target). Losses appear smaller than they would from a BF16 master, because the first pass already removed most of the information. That is a property of this master, not evidence of a better recipe.
4. PPL is not generation quality. It measures language-modelling loss on one English corpus. The 1-bit tiers pass the numeric gate but have not been evaluated for instruction following, long-context behaviour, or agentic use. Test before deploying.
5. No benchmark suite was run. No MMLU, GSM8K, or coding evaluations — only perplexity.

License

MIT, inherited from the upstream model. See the original repository for terms.

---

Quantized by the 6block team.