deepseek-v4-flash-sm120-recipe

DeepSeek-V4-Flash-DSpark on 4× RTX PRO 6000 (sm120) — stock vLLM 0.25.1

Deployment recipe (no weights hosted here). Full repo, scripts, benchmarks, and docs → https://github.com/hermia-ai/deepseek-v4-flash-sm120-stock-vllm

deepseek-ai/DeepSeek-V4-Flash-DSpark (FP8 + MXFP4, ~157 GB) serving on 4× RTX PRO 6000 Blackwell (sm120) with stock vLLM 0.25.1 + flashinfer-python 0.6.14 — no fork, no patch, no Docker. Coherent, full OpenAI + Anthropic tool calling, ~133 tok/s, 256K context.

The whole recipe

pip install "vllm==0.25.1" "flashinfer-python==0.6.14"
export FLASHINFER_DISABLE_VERSION_CHECK=1 NCCL_P2P_DISABLE=1
vllm serve /path/to/DeepSeek-V4-Flash-DSpark \
  --tensor-parallel-size 4 --enable-expert-parallel --kv-cache-dtype fp8 \
  --max-model-len 262144 --gpu-memory-utilization 0.90 \
  --kernel-config '{"moe_backend":"marlin"}' \
  --enable-auto-tool-choice --tool-call-parser deepseek_v4 --reasoning-parser deepseek_v4 \
  --trust-remote-code

The MXFP4 experts need moe_backend=marlin (deep_gemm_mega_moe is SM100-only).

Honest limits of the stock path

• Single-request context caps at ~256K. A prefill past that crashes with a DeepGEMM 719 (CUDA_ERROR_LAUNCH_FAILED); it's not chunk size, and VLLM_USE_DEEP_GEMM=0 won't boot. The KV pool holds 1M — it's a kernel launch limit, not memory.
• No speculative decode on stock — MTP/DSpark die at init (num_tokens > 64 ... sparse_mla_sm120_decode_dsv4).
• Both are lifted off-stock by vLLM PR #41834 (jasl) and community forks. Base sm120 support is upstream (#43477, merged).

Full traces, benchmarks, and the prior-art map are in the GitHub repo. Apache-2.0.