Downloads · 30 days
41
2% of all-time downloads
deepseek-v4-gguf
ivanfioravanti/deepseek-v4-gguf
deepseek-v4-gguf is a machine learning model from ivanfioravanti. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
Mixed-quantization GGUF for the DS4 runtime.
All-time downloads
2.7K
Public
Repo size
97.6 GB
Likes
7
Public
Repo makeup
Click a slice to open those files.
.gguf97.6 GB · 100%
At a glance
- License
- mit
- Access
- Public
- Created
- Jul 11, 2026
- Updated
- Jul 11, 2026
- SHA
- cb3f58ac
Base models
Model card
From the Hugging Face model README
DeepSeek V4 Flash — mixed Q2/Q4 GGUF
Mixed-quantization GGUF for the DS4 runtime.
Quantization recipe
- Q4_K routed experts in layers
10,30, and39–42. - Other routed-expert layers use the Q2 imatrix recipe: IQ2_XXS gate/up and Q2_K down tensors.
- Attention projections, shared experts, and output tensors follow the DS4 Q2-imatrix template.
- Six of 43 layers use Q4 routed experts (
13.95%). - File size: 97,591,747,456 bytes (
90.89 GiB).
The selected Q4 tensors were copied byte-for-byte from the compatible Q4-imatrix GGUF. Other tensors remain byte-identical to the Q2-imatrix base.
Evaluation
All variants below were scored on the same 100 official DeepSeek V4 Flash continuations containing 2,289 target tokens.
| Model / routed-expert recipe | Size | Average NLL ↓ | NLL vs Q2 | Wins vs Q2 | First-token matches ↑ | Average greedy LCP ↑ |
|---|---|---|---|---|---|---|
| Q2 imatrix baseline | 80.76 GiB | 0.368177401 | — | — | 63 / 100 | 5.96 |
Antirez mixed recipe: Q4 layers 37–42 | 90.89 GiB | 0.340543836 | -7.506% | 67 / 33 | 68 / 100 | 6.90 |
This model: Q4 layers 10, 30, 39–42 | 90.89 GiB | 0.338330454 | -8.107% | 73 / 27 | 68 / 100 | 7.47 |
| Full Q4 imatrix | 153.33 GiB | 0.173978006 | -52.746% | 87 / 13 | 86 / 100 | 12.88 |
Lower NLL is better. First-token matches count how often greedy generation starts with the same token as the official continuation. Greedy LCP is the number of consecutive matching tokens before the first divergence, so higher is better.
At the same 90.89 GiB size as the Antirez six-layer recipe, this configuration improved NLL, preserved the same first-token match count, and increased average greedy LCP from 6.90 to 7.47.
Usage
./ds4 \
-m DeepSeek-V4-Flash-Layers10-30-39-42Q4KExperts-OtherExpertLayersQ2-imatrix.gguf \
--ctx 4096 \
-p "Hello"
This GGUF uses DS4-specific DeepSeek V4 tensor naming and mixed routed-expert quantization.
- License
- mit
- Created
- Jul 11, 2026
- Updated
- Jul 11, 2026