DeepSeek-V4-Flash-0731-GGUF

DeepSeek-V4-Flash-0731-GGUF (hetero-v2)

Ablation-guided 'heterogeneous' GGUF quantization of DeepSeek-V4-Flash-0731.

95.8 GB single file. Fits 128 GB memory, leaving ~32 GB for KV cache (~150K tokens) to enable multi-turn sessions.

Quantization schema

Quantization schema derived from the Deepseek V4 Pro model's quantization recommendations (hence, heterogeneous), refined with ablation signal analysis. I was curious if the models have been trained on and thus have 'internalized' their own architecture such that they could advise an effective distribution.

Three-tier expert quantization, ranked by confabulation-routing ablation signal from a differential heatmap over the model's own expert activation patterns:

Approach is verified by DSL architecture documents regarding MoE routing and attention architecture.

Layers 32 and 40 showed the highest confabulation-routing signal (routing decisions most likely to produce hallucinated output when degraded), so they get the most bits. Layers 13, 26, 36, 42 are the next tier. Everything else is sparse enough that Q2_K preserves quality.

How to run

Requires llama.cpp with DeepSeek-V4 support. Mainline llama.cpp gained V4 support in mid-2026; earlier builds may need the antirez fork.

llama-server \
  -m DeepSeek-V4-Flash-0731-hetero-v2.gguf \
  --ctx-size 4096 \
  --n-gpu-layers 99 \
  --port 8080

Minimum context size is 4096 due to MLA compressor bank working set requirements. 32,768 is enough for standard reasoning tasks.

Architecture

DeepSeek-V4-Flash is a Mixture-of-Experts transformer:

V4-specific features not in V3: hash routing tables (ffn_gate_tid2eid), MLA compressor banks, DSA indexer, hyper-connections (hc_*), attention sinks, grouped output projections.

Conversion pipeline

Two-phase conversion from the FP4/FP8 source weights:

1. Phase 1 (convert_hf_to_gguf.py): Dequantize FP4 experts and requantize to Q2_K/Q3_K/Q4_K by ablation tier. Base tensors (attention FP8, shared FP8, embeddings BF16) go to Q8_0.
2. Phase 2 (llama-quantize): Requantize attention and shared expert Q8_0 down to Q6_K. Expert tiers and embeddings/output preserved via tensor-type override file.

Performance notes

Tested on Apple M5 Max, 128 GB unified memory:

• Generation speed comparable to uniform-quant at same expert BPW (~30 t/s)
• Output quality at API parity on qualitative benchmarks (math, translation, code, explanation); see the v1 report for methodology. v2 refines the expert tiering but uses the same evaluation approach

Credits

• DeepSeek Labs for the incredible open model
• Salvatore Sanfilippo (antirez) for the original llama.cpp V4 fork and the --deepseek4-expert-outtypes conversion flag