Downloads · 30 days
783
19% of all-time downloads
takanori-ishikawa/DeepSeek-V4-Flash-0731-Pruned-155B-A13B-GGUF
DeepSeek-V4-Flash-0731-Pruned-155B-A13B-GGUF is a text generation model from takanori-ishikawa. Use it when you need the model to write or continue text. The card lists the license as mit.
Not a DeepSeek release. This is an unofficial, community-made expert-pruned derivative of DeepSeek-V4-Flash-0731, not affiliated with or endorsed by DeepSeek. This is a derivation chain (base → quantization → pruning)…
Downloads · 30 days
783
19% of all-time downloads
All-time downloads
4K
Public
Repo size
164 GB
Likes
0
Public
Click a slice to open those files.
.gguf164 GB · 100%
From the Hugging Face model README
Not a DeepSeek release. This is an unofficial, community-made expert-pruned derivative of DeepSeek-V4-Flash-0731, not affiliated with or endorsed by DeepSeek. This is a derivation chain (base → quantization → pruning), not a model merge.
The builds are sized to run fully GPU-resident on a 64 GB Apple Silicon machine with llama.cpp — concretely, a Mac mini M4 Pro 64 GB, the machine this build targets and was developed and measured on. Routed experts are reduced from 256 to 128 on the 40 backbone MoE layers; the first 3 MoE layers ("hash layers") keep all 256 experts and their original fixed routing tables. The result is ~155.5B total parameters with ~13B active per token, at IQ3_XXS quantization.
Capability check: on a 100-instance SWE-bench Verified subset, this build resolves 77.5% running agentically on that single Mac mini, averaged over two passes (76/100 and 79/100). Under the identical harness, DeepSeek's official API serving the unpruned FP8 reference weights resolves 84.0% (86/100 and 82/100) — a 6.5-point gap (details, comparison table, and caveats under Quality and limitations).
Quantized bytes are unmodified slices of unsloth's UD-IQ3_XXS shards: pruning slices the expert axis byte-for-byte, with no dequantize/requantize step, so no new quantization loss is introduced. Note that the imatrix calibration behind those bytes was computed for the unpruned model; whether it remains optimal after pruning has not been evaluated separately.
All figures measured by the author on the Mac mini M4 Pro 64 GB. This project started from running the unpruned model on the same machine: no unpruned quantization fits in 64 GB of RAM, so the weights stream from SSD via mmap and the best you can do without modification is partial GPU offload. Pruning is what makes the model fit fully on GPU — roughly 4–5× the decode and 3–4× the prefill of that best unmodified setup.
| Configuration | Decode | Prefill |
|---|---|---|
| This build + DSpark sidecar (patched llama.cpp, all-GPU) | up to 21.2 t/s short context; 14.3–16.6 t/s at 8k–32k+ agent context (band medians, 1,110 turns) | 99–126 t/s (12k context) |
| Unpruned base, best unmodified setup on the same machine (vanilla llama.cpp, partial GPU offload, mmap-streamed) | ~4 t/s | ~33 t/s |
Decode of the unpruned model barely moves with offload or quantization choice (3.3–4.1 t/s across configurations) because the weights that matter stream from disk; only making the model small enough to live entirely on GPU changes that picture.
| File | Size | What it is |
|---|---|---|
DeepSeek-V4-Flash-0731-Pruned-155B-A13B-UD-IQ3_XXS-*.gguf (2 shards) | 55 GiB | Main pruned build. Needs the llama.cpp patch below |
DeepSeek-V4-Flash-0731-Pruned-155B-A13B-ZP256-UD-IQ3_XXS-*.gguf (3 shards) | 97 GiB logical / ~63 GiB after re-sparsifying | Vanilla-compatible build. Runs on vanilla llama.cpp |
DeepSeek-V4-Flash-0731-DSpark-Draft-2.9B-IQ2_XXS.gguf | 1.1 GiB | Pruned DSpark draft model for speculative decoding (patched llama.cpp only) |
llama-cpp-DeepSeek-V4-Flash-0731-Pruned-155B-A13B.patch | 13 KB | llama.cpp patch: per-layer expert count loading |
punch_zero_holes.py | — | Re-sparsify tool for the ZP256 build (macOS/APFS) |
Main build (patched llama.cpp) — best speed. The hash layers keep 256 experts while backbone layers have 128, so the loader needs a small patch (5 files, +54/−15 lines) that reads a deepseek4.hash_layer_expert_count metadata key. Vanilla llama.cpp rejects these files with:
llama_model_load: error loading model: check_tensor_dims: tensor 'blk.0.ffn_gate_inp.weight'
has wrong shape; expected 4096, 128, got 4096, 256
If you see that error, you are on an unpatched build.
ZP256 build (vanilla llama.cpp, or other GGUF runtimes that run the base model) — structurally identical to the base 256-expert model. Instead of physically removing experts, the router bias (exp_probs_b) of non-selected backbone experts is set to a large negative value so they are never routed to, and their weight slabs are zeroed. It should load anywhere the base GGUF loads (validated on vanilla llama.cpp; verify on other runtimes before relying on it). The catch: the file is 97 GiB logically, so fitting a 64 GB machine relies on mmap lazy loading plus keeping backbone experts on CPU (llama.cpp --override-tensor), which is considerably slower (~7.8 t/s decode vs ~17 t/s for the main build).
Build llama.cpp from the branch/tag DeepSeek-V4-Flash-0731-Pruned-155B-A13B-v1 — upstream master with the single loader commit applied — or apply the bundled .patch to a recent upstream checkout (validated against upstream commit fc35562ba; on much newer checkouts the patch may need trivial rebasing):
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git apply /path/to/llama-cpp-DeepSeek-V4-Flash-0731-Pruned-155B-A13B.patch
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j
On a 64 GB machine, raise the GPU wired-memory limit before serving — the macOS default (~75% of RAM) is not enough for the ~55 GiB model plus KV and compute buffers. The setting resets on reboot:
sudo sysctl iogpu.wired_limit_mb=61440
Even with the raised limit, the practical ceiling is ~60 GiB of total Metal buffers; configurations that exceed it (for example -ub 2048 with the sidecar loaded) fail with a runtime Metal OOM.
Serve fully GPU-resident with the speculative-decoding sidecar (the author's long-running local server configuration on the Mac mini M4 Pro 64 GB):
./build/bin/llama-server \
-m DeepSeek-V4-Flash-0731-Pruned-155B-A13B-UD-IQ3_XXS-00001-of-00002.gguf \
-c 196608 --parallel 2 -ngl 99 -ub 512 \
--fit off --no-repack \
-md DeepSeek-V4-Flash-0731-DSpark-Draft-2.9B-IQ2_XXS.gguf \
--spec-type draft-dspark -ngld 99 --spec-draft-n-max 1 \
-ctkd q8_0 -ctvd q8_0 \
--jinja
Pass the first shard as -m; the loader picks up the rest automatically.
Configuration notes, all from measurement on the machine above:
-ctk/-ctv q8_0): the DeepSeek-V4 attention takes a slow path and decode drops roughly in half (16.5 → 9.3 t/s). Quantizing the draft KV (-ctkd/-ctvd q8_0) is fine and has no measurable effect on acceptance.-ub 512 is required when the sidecar is loaded on 64 GB: with -ub 2048 the combined compute buffers exceed the practical wired-memory ceiling (~60 GiB of Metal buffers) and the server hits a runtime Metal OOM. The prefill cost of the smaller batch is 10–20%. If you drop the sidecar, -ub 2048 is the better prefill setting.Long unattended agentic runs (hours of SWE-bench-style rollouts) are markedly more stable with three adjustments, all found the hard way on this build:
function tool type, logging unsupported Responses tool type '...' skipped. Codex declares its multi-agent tool group as a namespace-type tool and web search as a web_search-type tool; both get dropped, so the model sees a smaller tool set than Codex assumes. Set top-level web_search = "disabled" and [features] multi_agent = false in the Codex config. Top-level keys must appear before any [table] header in the TOML, or they are silently ignored.--parallel 1 -c 262144 (256K). Long rollouts stop hitting per-slot context truncation, and Codex compacts context far less often; set Codex's model_context_window to the slot size.unsupported Responses tool type warnings — new tool types show up there first.A rare failure mode to know about: on a small fraction of first turns (roughly 0.1–0.3% in our runs before these adjustments) the model narrates a whole session instead of calling tools, emitting orphan </think> closers until it fills the slot. There is no output cap by default, so one runaway generation can burn the full slot (~90 minutes at 256K). It shows up in the server log as truncated = 1 with n_tokens near the slot size.
llama-server \
-m DeepSeek-V4-Flash-0731-Pruned-155B-A13B-ZP256-UD-IQ3_XXS-00001-of-00003.gguf \
-c 32768 -ngl 99 --jinja \
--override-tensor 'blk\.([3-9]|[1-3][0-9]|4[0-2])\.ffn_(gate|up|down)_exps\.weight=CPU'
The --override-tensor pattern keeps the backbone expert tensors on CPU, which is what makes a 64 GB machine workable (~7.8 t/s decode). With enough unified/system memory you can drop it and load everything on GPU. Do not combine the ZP256 build with the speculative sidecar: with experts on the CPU path, drafting is bandwidth-bound and counterproductive.
The zeroed expert slabs make ~42 GiB of the ZP256 shards pure zeros. On macOS/APFS you can turn those regions back into filesystem holes after download, shrinking physical usage from 97 GiB to ~63 GiB without changing file contents:
python3 punch_zero_holes.py DeepSeek-V4-Flash-0731-Pruned-155B-A13B-ZP256-UD-IQ3_XXS-*.gguf
Note that ordinary copies (rsync, scp, cross-volume cp) re-materialize the holes; within one APFS volume use cp -c.
DeepSeek-V4-Flash-0731-DSpark-Draft-2.9B-IQ2_XXS.gguf is a pruned build of the official DSpark draft model: 32 experts, IQ2_XXS expert quantization, 1.1 GiB. Because the DSpark draft conditions on the target model's hidden state, its top-1 agreement tolerated expert pruning well in the author's measurements: this K32 build's acceptance rate was within run-to-run noise of the unpruned draft on the workloads tested (not a benchmark-grade claim).
Measured effect on the machine and server configuration above (--spec-draft-n-max 1):
--spec-draft-n-max 1 is the fastest setting on this machine, not a conservative fallback: per-position acceptance decays (~0.75 → ~0.55 → ~0.39) while Metal's small-batch verify cost grows almost linearly with batch size, so only the first draft position pays for itself. Measured across context lengths, n_max 2 gains less than n_max 1 and n_max 3 is a net loss. On backends where small-batch verify is closer to flat cost (e.g. CUDA), larger n_max may come out ahead — re-measure before porting this setting. On M5-generation Apple Silicon, reports so far show the Metal 4 tensor units engaged only for large-batch matmul, with measured gains limited to prefill, so the small-batch verify path — and with it this ordering — is not expected to change there for nowThe sidecar requires the patched build and the main (non-ZP256) model, fully GPU-resident.
n_expert 128 + hash_layer_expert_count 256) is exactly what the loader patch implements.The GGUF general.* metadata (model name, source URLs) intentionally retains the upstream quantization's values, so inspection tools will report the original model identity; the file names and this card are the authoritative description of what the files contain.
The subset is 100 instances stratified by repository: an original 50, plus 50 drawn from the complement so that the original 50 stay fully contained. Both passes cover all 100: 76/100 = 76.0%, Wilson 95% CI [66.8, 83.3] on the first, and 79/100 = 79.0%, CI [70.0, 85.8] on the second. The pass@1 mean is 77.5%.
For scale, the same 100 instances recomputed from public per-instance leaderboard data, which is single-pass and so compares against this build's first pass (per_instance_details.json under evaluation/bash-only in SWE-bench/experiments, mini-SWE-agent submissions):
| Model | Same 100 instances | Full 500 |
|---|---|---|
| GLM 5 (high) | 78/100 = 78% | 72.8% |
| MiniMax M2.5 (high) | 75/100 = 75% | 75.8% |
| This build, on one Mac mini M4 Pro 64 GB | 76/100 = 76% | — |
| DeepSeek V3.2 (high) | 72/100 = 72% | 70.0% |
| Kimi K2.5 (high) | 70/100 = 70% | 70.8% |
| GLM 4.6 (T=1) | 58/100 = 58% | 55.4% |
| Qwen3-Coder 480B/A35B | 55/100 = 55% | 55.4% |
Measurement conditions:
| Item | Value |
|---|---|
| Sample | 100 of SWE-bench Verified, repository-stratified random, fixed seed |
| Agent | Codex CLI, k=1, 120-minute cap per instance |
| Passes | 2 over all 100 |
| Model | this IQ3_XXS build with the DSpark speculation sidecar, 256k-token context |
| Network | external access blocked (asserted at run start) |
Caveats: the leaderboard rows use a different agent harness (mini-SWE-agent vs Codex CLI here), so read that table as a sanity anchor, not a strict head-to-head ranking. For the degradation relative to the reference weights under this exact harness, see below.
The unpruned reference was measured under the identical harness: same 100 instances, same Codex CLI configuration and per-instance time cap, same network isolation (the agent container reaches only the model API through a relay; external access asserted blocked at run start and observed blocked in-run). The reference is DeepSeek's official API serving the release weights, which are FP8 e4m3 — no BF16 release exists, so FP8 is the cleanest available baseline. Input volume and cache-hit rates matched closely between the pass-1 and API runs (67–70M input tokens, ~98% cache hit), supporting that both sides processed equivalent context.
| Model | Pass 1 | Pass 2 | pass@1 mean |
|---|---|---|---|
| DeepSeek official API (FP8 reference, unpruned) | 86/100 = 86.0% | 82/100 = 82.0% | 84.0% |
| This build (pruned + IQ3_XXS, one Mac mini) | 76/100 = 76.0% | 79/100 = 79.0% | 77.5% |
The mean is 6.5 points below the reference; the per-pass gaps are 10 points on the first and 3 on the second.
Every cell is a single k=1 draw; compare cells only k=1 against k=1. In particular, do not read the union of two passes (pass@2) against a single run — that would be best-of-2 versus best-of-1.
Per pass, paired:
| Pass | Gap | This build only | Reference only | Exact two-sided p |
|---|---|---|---|---|
| 1 | 10 points | 0 | 10 | 0.002 |
| 2 | 3 points | 4 | 7 | 0.549 |
Read the delta as the combined cost of expert pruning, IQ3_XXS quantization, and serving-stack differences (llama.cpp vs DeepSeek's API implementation), not the pruning delta alone. Known residual asymmetries, all minor: tool-call parsing is llama.cpp's on the local side and DeepSeek's own on the API side; the API executes parallel tool calls unconditionally while the local build declares them unsupported; sampling runs at each side's upstream defaults.
Claims are limited to what was measured, on coding-agent workloads:
Broader evaluations are limited by the single consumer machine this project runs on — every suite that needs an unpruned-base rerun doubles the GPU time, so a full benchmark-suite comparison is not realistic. Apart from the SWE-bench subset above, direct Japanese-language comparison against base and further standardized benchmarks have not been completed. Treat this as a workload-specific build, not a general-purpose replacement for the base model.
See the bundled LICENSE file for the full notices.
Work that directly shaped this build:
Surveyed MoE-pruning alternatives (not adopted; names searchable on arXiv, unlinked here to keep this repo off those papers' citation listings):
The pruning/selection pipeline consumes the base GGUF shards plus recorded expert-selection IDs, and is deterministic byte-slicing from there. The loader patch is one commit on top of upstream llama.cpp; the branch and tag above pin the exact revision it was validated against.