DeepSeek-V4-Flash-Vision-Exp-NVFP4

DeepSeek-V4-Flash-Vision-Exp-NVFP4

NVFP4 (W4A4) conversion of deepseek-ai/DeepSeek-V4-Flash-Vision-Exp (MIT license), scoped to the routed experts only (all 11,008 expert linears across 43 layers × 256 experts). Attention (MLA), router, shared experts, MTP layers, embeddings, and the vision tower are preserved verbatim in their original precision.

How this was made

Not a fresh calibration quant: DeepSeek's canonical checkpoint already stores routed experts in MXFP4. This repo is a lossless transcode of those expert weights into the standard NVFP4 block-scale layout that vLLM's FP4 MoE kernels consume (modelopt format), produced with NVIDIA TensorRT Model Optimizer tooling. Weight bytes verified byte-exact against NVIDIA's official nvidia/DeepSeek-V4-Flash-NVFP4 recipe per expert; the only borrowed values are per-layer input_scale priors from the same reference (measured effect below lane noise). Transcode checks: zero lossy tensors.

Validation (stock vllm/vllm-openai:v0.28.0, 2×B200, TP=2)

First vLLM boot of Vision-Exp. Greedy decoding, four-state classifier:

The steered-delivery delta against the older HF-lane baselines is a lane effect (fp8 KV cache + kernel set), not quant damage: the FP8 control on the identical vLLM stack lands at the same place as this NVFP4 (±1 item, CIs overlap). Per-item agreement with FP8: stock 29/32; steered flips are symmetric borderline noise. Zero garbled outputs anywhere. Serving scoreboard: weightless BENCHMARK.md and the README lane table.

Serving

vllm serve msuiche/DeepSeek-V4-Flash-Vision-Exp-NVFP4 \
  --tensor-parallel-size 2 --kv-cache-dtype fp8   # B200 validated

• Validated: sm100 (B200), flashinfer::trtllm_fp4_block_scale_moe kernel.
• GB10 (DGX Spark): goes through the modelopt_gb10_hybrid plugin, which expects exactly this W4A4 layout (see weightless recipes).
• H100 (sm90): no native W4A4 MoE kernel for this shape; would need moe_backend="marlin" (W4A16, drops activation scales) — untested.
• Vision tensors: stock vLLM's DeepseekV4ForCausalLM is text-only and refuses vision.*/aligner.*/image_* plus the vision-routing biases. strip_vision.py (included in this repo) produces a text-only variant that boots on stock vLLM today; use the full artifact on a vision-capable runtime.

Steering

Compatible with the weightless GLP control vectors — see msuiche/DeepSeek-V4-Flash-Vision-Exp-abliterated-cyber-GLP-29 (apply via the weightless vLLM hotfix). Quantisation does not degrade a direction; verify your lane's delivery against the table above rather than the older HF-lane numbers.

License

MIT (inherited from the base model). Copyright (c) 2023 DeepSeek.