DeepSeek-v4-flash-180b-W4A16-VOLTA-Ampere

DeepSeek-V4-Flash-180B — W4A16 (REAP K160)

<p align="center"> <img src="israeliAI.png" alt="Israeli AI" width="420"/> </p>

A REAP-pruned, W4A16-quantized DeepSeek-V4-Flash that runs on pre-Hopper NVIDIA GPUs — Volta (sm_70) and Ampere (sm_80/86) — hardware with no native FP8/FP4 and no official DeepSeek-V4 support, via a bundled SM70/pre-Hopper Triton port. Verified on 4× V100 32 GB. Also runs stock on Hopper/Blackwell (sm_90+). Needs ~100 GB aggregate VRAM.

• Total params: ~180B (K160 REAP prune of the 284B base)
• Active / token: 6 routed experts + 1 shared (of 160 kept)
• Format: compressed-tensors W4A16 (int4, symmetric, group_size 128, pack-quantized); attention/indexer/compressor/shared-experts kept BF16
• On-disk: ~97 GB, 10 safetensors shards
• Arch: deepseek_v4 — 43 layers, hidden 4096, sparse MLA + lightning indexer, hybrid SWA/compressed attention, native ctx 1,048,576
• Hermes/OpenClaw: tested, production validation; clarify 90% is absolute battery score

What this is (lineage)

deepseek-ai/DeepSeek-V4-Flash  (284B MoE, 256 routed experts)
  └─ BF16 upcast:        RedHatAI/DeepSeek-V4-Flash-BF16
     └─ REAP prune K160: keep 160/256 experts per layer (Cerebras REAP keep-maps),
        router/hash tables remapped  →  bit-identical to 0xSero/DeepSeek-V4-Flash-180B
        on the hash-routed layers (0-2)
        └─ W4A16 g128:   data-free RTN, routed experts int4; attention kept BF16
           └─ THIS MODEL

• REAP = Router-weighted Expert Activation Pruning (Cerebras Research, arXiv:2510.13999). One-shot MoE compression: keep the experts the router actually depends on, delete the cold ones, remap the tables. No retraining.
• The keep-maps come from the published REAP observation set; layers 0–2 use hash routing and reuse the exact remapped tid2eid tables, so routing on those layers is identical to the reference 180B checkpoint.

Quality (as-served, W4A16, 4K ctx)

Production-validated: this checkpoint runs as the live primary brain of a tool-calling agent (reasoning + native tool calls + multi-turn), not just offline benchmarks.

Objective ground-truth battery + stability + retrieval:

The 90% is the model's absolute score on our objective battery, not a retention figure vs the unpruned base (which we did not run). Treat it as "scores 90% on these tasks," not "90% of the original."

Tool-calling works out of the box with --enable-auto-tool-choice --tool-call-parser deepseek_v4 (DSML format): correct tool selection, valid JSON args, multi-turn tool-result round-trips, and no spurious calls on plain questions. Tool prompts show lower derailment than terse ones.

Recommendations: run with thinking enabled (suppresses the derailment tail to ~0; costs more output tokens, not lower tok/s). For agent/tool use, wrap tool-call steps in a retry/validate loop — a small per-step tail compounds over long chains. This is a pruned derivative: evaluate on your own tasks before production.

---

Deployment

Two paths depending on your GPU generation.

A. Hopper / Blackwell (sm_90+) — stock

The weights are standard deepseek_v4 + compressed-tensors W4A16. Serve with an upstream/1Cat vLLM build that supports DeepSeek-V4, no patches:

vllm serve DeepSeek-V4-Flash-W4A16 \
  --tensor-parallel-size <N> \
  --reasoning-parser deepseek_v4 \
  --enable-auto-tool-choice --tool-call-parser deepseek_v4

B. Volta / Ampere (pre-Hopper, sm_70–sm_86) — requires the bundled port

Stock vLLM's deepseek_v4 path is FP8/FP4 + FlashMLA-sparse + CUTLASS-DSL — none of which exist on pre-Hopper GPUs. The port engages automatically for device_capability.major < 9 (Volta and Ampere) and adds a pure-Triton path:

Tested on 4× V100 (sm_70). Ampere (A100 sm_80, RTX 3090 sm_86) uses the identical arch-independent Triton path and should work, but was not verified here — validate before relying on it.

• BF16/FP16 split-K sparse-MLA attention kernel (no FlashMLA)
• bit-exact e4m3 emulation in Triton (Volta Triton has no fp8e4nv)
• Triton indexer MQA-logits kernels (replacing DeepGEMM, sm_90+ only)
• torch/Triton fallbacks for the mHC hyper-connection kernels (incl. a fused Sinkhorn kernel)
• expert-count-agnostic router fallback (REAP's 160 experts aren't templated in the fused CUDA router)

Install:

# 1) a 1Cat-vLLM venv that already registers DeepseekV4ForCausalLM (with SM70 quant support)
# 2) apply the port:
VLLM=/path/to/venv/lib/python3.12/site-packages/vllm bash sm70-port/apply-port.sh

Serve (4× V100 32 GB — see serve-dsv4-180b.sh). Non-obvious but required:

export CUDA_DEVICE_ORDER=PCI_BUS_ID
export CUDA_VISIBLE_DEVICES=<your 4 V100 indices>
export NCCL_P2P_DISABLE=1                     # NVLink P2P OOMs at these weight sizes
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
export VLLM_SM70_QUANT_BACKEND=turbomind      # route W4A16 to turbomind, not marlin
export VLLM_SM70_COMPRESSED_TENSORS_TURBOMIND=1

vllm serve DeepSeek-V4-Flash-W4A16 \
  --tensor-parallel-size 4 --disable-custom-all-reduce \
  --gpu-memory-utilization 0.93 \
  --kv-cache-memory-bytes 1395864371 \        # MUST pin: weights ~26.5GiB/GPU leave little room
  --max-model-len 4096 --max-num-seqs 2 --max-num-batched-tokens 2048 \
  --kv-cache-dtype fp8_e4m3 \
  --reasoning-parser deepseek_v4 \
  --enable-auto-tool-choice --tool-call-parser deepseek_v4 \
  --compilation-config '{"cudagraph_mode":"full_and_piecewise","cudagraph_capture_sizes":[1,2]}'

Gotchas (learned the hard way):

• Pin --kv-cache-memory-bytes. Weights are ~26.5 GiB on a 32 GiB card; without pinning, either the KV check fails or Triton kernel launches OOM. Adjust the value to leave ~2 GiB free per GPU.
• Driver must be the proprietary NVIDIA driver (open module can't drive Volta — no GSP). CUDA 12.x + Triton.
• Perf: ~16 tok/s single-stream, ~21 tok/s aggregate @ 4 concurrent, on 4× V100. Decode is compute-bound at batch 1 (≈15× under the HBM2 roofline), so it scales with concurrency. --enforce-eager is ~2.5× slower — keep CUDA graphs on.
• Do not W4A16-quantize the attention on this port: the turbomind attention gemm is incompatible with full-graph capture (illegal memory access), and it's a net slowdown anyway (compute-bound).
• If a request returns CJK/repetition garbage under concurrency, that's a decode-path bug class — this port fixes the known one; report others with the exact batch shape.

---

Reproduce the pipeline

pipeline/ contains the end-to-end recipe:

• prune_dsv4_reap.py — REAP prune of the BF16 base using reap_plan_k160.json keep-maps
• quant-dsv4-180b.py — streaming RTN W4A16 g128 pack-quantize (routed experts only)
• quality_eval.py, quality_eval2.py — the objective eval harness above

License & credits

MIT for the recipe/tooling; base weights follow the DeepSeek-V4-Flash license (review before use).

• Base: deepseek-ai/DeepSeek-V4-Flash; BF16 upcast: RedHatAI/DeepSeek-V4-Flash-BF16
• REAP: Lasby et al., REAP the Experts (arXiv:2510.13999), Cerebras Research
• Keep-map lineage: 0xSero/DeepSeek-V4-Flash-180B
• Serving: 1Cat-vLLM fork + the bundled SM70 port

This is an experimental derivative, not the full DeepSeek-V4-Flash. Validate before trusting it in production.