Downloads · 30 days
1.7K
9% of all-time downloads
Rednalreden/DeepSeek-V4-Flash-0731-dwarfstar-q2-gguf
DeepSeek-V4-Flash-0731-dwarfstar-q2-gguf is a text generation model from Rednalreden. Use it when you need the model to write or continue text. It is set up for gguf. The card lists the license as mit.
A 2-bit, importance-matrix-guided GGUF quantization of deepseek-ai/DeepSeek-V4-Flash-0731, built with the DwarfStar (ds4) toolchain.
Downloads · 30 days
1.7K
9% of all-time downloads
All-time downloads
19.4K
Public
Repo size
86.7 GB
Likes
15
Public
Click a slice to open those files.
.gguf86.7 GB · 100%
From the Hugging Face model README
A 2-bit, importance-matrix-guided GGUF quantization of
deepseek-ai/DeepSeek-V4-Flash-0731,
built with the DwarfStar (ds4) toolchain.
Intended for 96 GB and 128 GB machines that cannot hold a 4-bit build.
| File | DeepSeek-V4-Flash-0731-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-imatrix.gguf |
| Size | 86,720,111,520 bytes (80.8 GiB) |
| Resident model | 80.76 GiB |
| Context overhead | ~0.86 GiB at --ctx 32768 |
| Architecture | 43 layers, 256 routed experts (top-6), 1 shared expert |
| Tensor group | Type |
|---|---|
Routed experts, gate (w1) / up (w3) | IQ2_XXS |
Routed experts, down (w2) | Q2_K |
| Attention projections | Q8_0 |
| Shared experts | Q8_0 |
| Output | Q8_0 |
| HC / compressor / indexer | F16 |
This matches the established DwarfStar Flash q2-imatrix recipe, applied to the
0731 checkpoint. The 0731 weights ship as fp4 routed experts (I8 packed E2M1
with F8_E8M0 block-32 scales) and fp8 dense tensors; deepseek4-quantize
dequantizes both to f32 before requantizing.
Activation statistics were collected with the ds4 runtime itself, hooking the
layer-major Metal prefill graph, so the counts come from the real DS4 inference
graph rather than a proxy implementation.
gguf-tools/imatrix/dataset/rendered_prompts.txt (DS4-rendered chat
prompts: source review, long context, agent/tool-call DSML, prose tasks, and
benchmark prompts, in both thinking and non-thinking forms)For gate/up the collector records squared FFN-normalized input activation; for down, the squared route-weighted SwiGLU row.
Scored against 100 official DeepSeek V4 Flash continuations by target-token negative log likelihood, comparing this build to an otherwise identical build of the same checkpoint quantized without an imatrix:
| Build | avg NLL |
|---|---|
| 0731 Q2, no imatrix | 0.579809 |
| 0731 Q2, imatrix (this file) | 0.571552 |
| Relative change | -1.42% |
Per-case wins: 54 imatrix / 46 no-imatrix.
Both builds are the same checkpoint, so this isolates the imatrix contribution. Note that the reference continuations were collected from an earlier Flash checkpoint, which raises the absolute NLL for both builds equally; only the relative difference is meaningful here.
git clone https://github.com/antirez/ds4
cd ds4 && make
./ds4 -m DeepSeek-V4-Flash-0731-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-imatrix.gguf \
-p "Write a C function that reverses a string in place."
Or serve it:
./ds4-server -m DeepSeek-V4-Flash-0731-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-imatrix.gguf --ctx 100000
Observed on an Apple M5 Max (128 GB, Metal): prefill 86.9 t/s, generation 29.6 t/s at short context. Throughput falls as context grows.
Q8_0, and the HC/compressor/indexer path stays at F16.ds4 runtime.MIT, inherited from the base model.