deepseek-v4-tiny-random-nvfp4

DeepSeek-V4 Tiny Random NVFP4

This is a randomly initialized, tiny debugging checkpoint with the DeepSeek-V4 architecture. It is not a trained language model and must not be used to evaluate model quality.

It was converted from yujiepan/deepseek-v4-tiny-random at revision 082d9c80d8e5b3f1b81e4a18e0a22daad84537af to exercise NVFP4 fused-MoE kernels on NVIDIA Blackwell GPUs.

Quantization layout

• All routed expert w1, w2, and w3 weights in the 7 transformer layers were converted from MXFP4 group-32 to NVFP4 group-16.

• The routed-expert tensors use the ModelOpt-compatible NVFP4 schema: packed uint8 weights, FP8 E4M3 block scales, and scalar global/input scales.

• Dense FP8 tensors, shared experts, embeddings, the LM head, and MTP tensors are unchanged from the source mock checkpoint.

• The model config uses activation="silu" and swiglu_limit=10.0, requiring the clamped SwiGLU semantics:

  gate = min(gate, 10.0)
  up   = clamp(up, -10.0, 10.0)
  out  = silu(gate) * up
  

This is therefore a mixed checkpoint intended specifically for routed-expert NVFP4 testing, not a claim that every linear layer is NVFP4.

Validation

The checkpoint was tested on an NVIDIA GB10 (SM121) with:

• vLLM nightly v0.26.1rc1.dev306+gcb8104839
• FlashInfer 0.6.18 plus the B12X SiLU clamp fix
• forced FLASHINFER_B12X NVFP4 MoE backend
• activation="silu", swiglu_limit=10.0

All seven B12X MoE wrappers were constructed with the limit and an end-to-end prefill plus two decode steps completed. The tested vLLM nightly predates the B12X clamp capability declaration and parameter forwarding, so it required an in-process adapter bridge. This is a runtime integration limitation, not part of the checkpoint.

Some non-MoE DeepSeek-V4 tiny-shape paths also required reference fallbacks on GB10. The checkpoint should be treated as an expert-kernel/debugging fixture, not as a drop-in production serving model.

Integrity

model.safetensors SHA256:

9859140eff50a9dbb0454aa5d008418dfbdbccb2dce3a5858d252bb125aa37b3

See CONVERSION_INFO.json for the source revision and conversion summary.

Limitations

• Random weights; generated text is meaningless.
• Intended for architecture, loader, quantization, and fused-MoE testing only.
• Requires software with DeepSeek-V4 and ModelOpt-style mixed-precision checkpoint support.
• B12X execution requires an SM120/SM121 GPU and a FlashInfer version that applies swiglu_limit to standard SiLU.