deepseek-v4-nvfp4-fp8

DeepSeek-V4-Flash NVFP4 + FP8, with DSpark MTP stages

Experimental. Do not use this checkpoint for evaluation.

It loads, serves, and resolves its kernels on GB10 (sm_121), and the kernel parity suite passes — but generation is not yet correct. Output is fluent and wrong: it answers arithmetic and factual prompts confidently with wrong values. The cause is under investigation and is currently believed to be in this checkpoint's quantization rather than in the serving code, because the sibling appmana/deepseek-v4-int4-int8 checkpoint generates correctly through the identical attention kernels and KV cache on the same hardware and the same build.

No benchmarks are published here, and none should be quoted from it.

NVFP4 routed experts with FP8 dense layers and an fp8_ds_mla KV cache, targeting NVIDIA GB10 (DGX Spark, sm_121) and consumer Blackwell.

What this is

The source is nvidia/DeepSeek-V4-Flash-NVFP4 with the three DSpark MTP draft stages from DeepSeek-V4-Flash spliced in, so a single checkpoint carries both the backbone and its speculative-decoding draft.

The MTP blocks remain MXFP4 — they are deliberately absent from the quantization_config.quantized_layers table, so they load under the base expert format rather than as NVFP4. A loader that applies one quantization format to the whole model will load them wrongly; per-module resolution is required.

Serving

Requires the AppMana vLLM fork (branch appmana/vllm-consumer-nvidia-platforms) — upstream vLLM's DeepSeek-V4 sparse-MLA path assumes Hopper-or-newer kernels. Kernels come from forks-sparkinfer (CuTe-DSL compressed MLA, NSA indexer, fused NVFP4 MoE) and forks-flash-mla-ampere-dsv4.

Kernel selection is checkpoint-driven — the vllm block in config.json:

"vllm": {
  "kernels": [
    "vllm.models.deepseek_v4.nvidia_sm12x.kernels.sparkinfer_sparse_mla_decode",
    "vllm.models.deepseek_v4.nvidia_sm12x.kernels.sparkinfer_sparse_mla_extend"
  ],
  "cache_type": "fp8_ds_mla"
}

Two ranks, one GB10 each:

vllm serve appmana/deepseek-v4-nvfp4-fp8 \
  --tensor-parallel-size 2 \
  --distributed-executor-backend mp \
  --tokenizer-mode deepseek_v4 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.68 \
  --kv-cache-memory-bytes 8589934592 \
  --speculative-config '{"method": "dspark", "num_speculative_tokens": 5, "draft_sample_method": "greedy"}'

Do not pass --attention-backend (it routes away from sparkinfer) or --kv-cache-dtype (it overrides the checkpoint's fp8_ds_mla). Confirm the vllm kernels resolved: ... line in the log before trusting any output.

Use the chat template. Raw /v1/completions prompts against this checkpoint degrade; /v1/chat/completions with --tokenizer-mode deepseek_v4 is the intended path.

On a unified-memory part the GPU allocation is host RAM, and --kv-cache-memory-bytes adds to the --gpu-memory-utilization budget rather than coming out of it; both together must leave headroom for the host.

Related

• appmana/deepseek-v4-int4-int8 — INT4 W4A16 experts + INT8 W8A16 dense, int8_ds_mla cache. Validated and benchmarked; serves a 1M-token context on twelve 24 GB Ampere GPUs. Use this one for anything real.