DeepSeek-V4-Flash-GGUF

A Q2_K version of https://huggingface.co/ssweens/deepseek-ai__DeepSeek-V4-Flash-GGUF-YMMV

---

---

---

---

---

🧪 Experimental GGUFs for DeepSeek-V4-Flash

A stopgap to experiment with DeepSeek-V4-Flash with CUDA and ROCm locally while the tools ecosystem catches up. Expect rough edges. Validated for text and coding coherence.

GGUF files for deepseek-ai/DeepSeek-V4-Flash.

⚠️ You need the custom fork

These GGUFs require a DeepSeek-V4-capable fork of llama.cpp. Vanilla llama.cpp doesn't support this architecture yet.

• llama.cpp fork: ssweens/llama.cpp-deepseek-v4
• Backends: Tested on CUDA and ROCm.

Performance

Example:

llama-server -ngl 99 --no-mmap -fa on -np 1 --reasoning-format auto --jinja --threads 3 -ts 4,4,3 -dev CUDA0,CUDA1,CUDA2 \
-m /mnt/supmodels/gguf/deepseek-ai__DeepSeek-V4-Flash/deepseek-ai__DeepSeek-V4-Flash-Q4_K_M.gguf -c 32768 -b 2048 -ub 512 -ctk q8_0 -ctv q8_0

Speed (custom, n=2)

Original model

• deepseek-ai/DeepSeek-V4-Flash

Thanks

• antirez — llama.cpp fork for Metal and CUDA in llama.cpp-deepseek-v4-flash
• ml-explore/mlx-lm #1192 — MLX DSV4 attention reference that informed the architecture work
• DeepSeek — open inference code and the technical report
• nisparks et al - some early implementation efforts and discussion
• llama.cpp — the project that makes local LLM inference possible