DeepSeek-V4-Flash-160t2-UD-IQ1_S

DeepSeek-V4-Flash-160t2 (UD-IQ1_S, per-layer expert pruning)

Hugging Face: siendsi/DeepSeek-V4-Flash-160t2-UD-IQ1_S

A 38% expert-pruned build of DeepSeek-V4-Flash-0731 (UD-IQ1_S), reduced from 82.4 GB → 51.0 GB (−38.2%) by keeping the 160 most-activated experts per layer (out of 256) and disabling the remaining 96.

This is a GGUF model for llama.cpp. It is a per-layer MoE expert pruning of the original — not a low-rank distillation and not a weight re-quantization. All non-expert weights and the kept experts are bit-identical to the original UD-IQ1_S checkpoint.

This is the optimal depth found in a pruning-depth study: keep-160 is the smallest build that stays at parity on the target profile. Going deeper (keep-144, −43%) starts to break reasoning tasks.

What was done

• Method: per-layer expert pruning. For each of the 43 MoE layers, the 160 experts with the highest activation counts (from a calibration "heat" map) are kept; the 96 least-used experts are disabled.
• Calibration heat: tuned to a target profile — Russian + English + coding + planning + agentic tasks, plus a math corpus (math is critical for agentic/reasoning tasks). The original calibration corpus (which included Japanese/Chinese) was not used, because it pruned experts that matter for the target profile.
• Quantization: unchanged — UD-IQ1_S (Unsloth dynamic quant).
• Architecture: deepseek4 (llama.cpp).

Why per-layer? DeepSeek-V4 expert IDs are per-layer — expert N in layer 0 is a different expert than expert N in layer 3 (0/256 shared). A global top-K expert map is therefore meaningless; pruning must be done independently per layer.

Size

Quality (A/B vs. original)

Evaluated against the original UD-IQ1_S on a 26-task target-profile battery (code / devops / multi-hop / OS / NLP / agent):

• keep-160: 24/26 parity. Two degradations, both on long, self-correcting reasoning tasks:
  1. dockerfile — the pruned model can loop in its reasoning (finish=length, empty content), whereas the original answers correctly.
  2. multi_hop_math — does not converge on a multi-step train problem.
• keep-144 (deeper): 23/26 — adds a third-class degradation on an agentic task (multi_tool). This is why keep-160 is the recommended depth.

Practical takeaway: for short, deterministic answers (knowledge, math, code snippets, NLP, OS, devops) this model is at parity with the original. For tasks that require long, self-correcting reasoning, keep-160 may fail to converge (it does not give wrong answers — it simply does not finish).

Speed (llama.cpp, -t 8)

Measured on keep-192 (same build pipeline, 60 GB); keep-160 is expected to be similar or slightly better on prefill (fewer expert weights to load):

Prefill is noticeably faster on the pruned model; generation is roughly at parity (the number of active top-k experts is the same, so the difference in weight loading is minimal).

Usage (llama.cpp)

llama-server \
  -m DeepSeek-V4-Flash-0731-UD-IQ1_S-00001-of-00003.gguf \
  -c 32768 -t 8 -fa on --load-mode mmap \
  --port 8087

Or with the llama CLI:

llama-cli -m DeepSeek-V4-Flash-0731-UD-IQ1_S-00001-of-00003.gguf \
  -c 32768 -t 8 -fa on --load-mode mmap

The model is multi-shard. Point llama.cpp at shard 00001; it will pick up the other shards automatically.

Files

Provenance

• Base: DeepSeek-V4-Flash-0731 (UD-IQ1_S, Unsloth dynamic quant)
• License: MIT (inherited from the base model)
• Pruning: per-layer top-160/256 experts, target-profile + math heat
• Built with: llama.cpp rebuild_gguf.py (per-layer prune config)
• Reproduction: see the glm-pruning project — DSV4_PRUNING_FINAL.md, tools/build_dsv4_prune_config_perlayer.py, tools/heat_target.py

Notes

• This is a pruned model, not a fine-tune. Behavior on the target profile is at parity with the original; out-of-profile (e.g. Japanese/Chinese) is not the goal and may be degraded.
• Very long contexts (>32k) were not stress-tested.
• A less-aggressive build (keep-192, 60 GB) is available at siendsi/DeepSeek-V4-Flash-192t2-UD-IQ1_S if you prefer the extra safety margin.