DeepSeek-R1-NVFP4-v2-mlpinf

Model Overview

Description:

This model is a quantized version of DeepSeek AI's DeepSeek R1 model, produced with the official NVIDIA TensorRT Model Optimizer DeepSeek recipe and calibrated with the MLPerf Inference deepseek-r1 calibration dataset. It is quantized with FP4 weights and activations (the same quantization scheme as nvidia/DeepSeek-R1-FP4-v2, including the wo module in attention layers) and is ready for inference with TensorRT-LLM. This model is ready for commercial/non-commercial use.

Third-Party Community Consideration

This model is not owned or developed by NVIDIA or DeepSeek AI. It has been produced by a third party from the open DeepSeek R1 checkpoint for the MLPerf Inference deepseek-r1 benchmark. See the original DeepSeek R1 Model Card.

License/Terms of Use:

MIT

Model Architecture:

Architecture Type: Transformers

Network Architecture: DeepSeek R1

Input:

Input Type(s): Text

Input Format(s): String

Input Parameters: 1D (One Dimensional): Sequences

Other Properties Related to Input: Context length up to 128K. Per DeepSeek's usage recommendations: temperature 0.5–0.7 (0.6 recommended), no system prompt, and for math problems ask for step-by-step reasoning with the final answer in \boxed{}.

Output:

Output Type(s): Text

Output Format: String

Output Parameters: 1D (One Dimensional): Sequences

Software Integration:

Supported Runtime Engine(s):

• TensorRT-LLM

Supported Hardware Microarchitecture Compatibility:

• NVIDIA Blackwell

Preferred Operating System(s):

• Linux

Model Version(s):

Quantized with NVIDIA TensorRT Model Optimizer, main branch @ 089c06e4 (July 2026), using the examples/deepseek recipe (deepseek_v3/ptq.py for calibration, deepseek_v3/quantize_fp8_to_nvfp4.sh for weight conversion).

Datasets:

Calibration Dataset: mlperf_deepseek_r1_calibration_dataset_500_fp8_eval — the official 500-sample MLPerf Inference deepseek-r1 calibration set. See the MLPerf deepseek-r1 reference README for dataset details and download instructions.

• Data collection method: Hybrid: Human, Automated
• Labeling method: Hybrid: Human, Automated

Inference:

Engine: TensorRT-LLM

Test Hardware: NVIDIA GB200 NVL72

Post Training Quantization

This model was obtained by quantizing the weights and activations of DeepSeek R1 to FP4 data type, ready for inference with TensorRT-LLM. Only the weights and activations of the linear operators within the transformer blocks are quantized (fine-grained FP4 block scaling with group size 16; the KV cache is quantized to FP8; MLA attention projections remain in higher precision). This optimization reduces the number of bits per parameter from 8 to 4, reducing the disk size and GPU memory requirements by approximately 1.6x.

Calibration runs the official recipe unmodified except for the calibration dataloader, which reads the MLPerf deepseek-r1 calibration set (500 samples, max sequence length 2048, batch size 4).

Usage

Deploy with TensorRT-LLM

To deploy the quantized checkpoint with TensorRT-LLM LLM API, use the following sample code (requires 8 GPUs of Blackwell generation or later, and TensorRT-LLM built from source with the latest main branch):

from tensorrt_llm import SamplingParams
from tensorrt_llm._torch import LLM

def main():

    prompts = [
        "Hello, my name is",
        "The president of the United States is",
        "The capital of France is",
        "The future of AI is",
    ]
    sampling_params = SamplingParams(max_tokens=32)

    llm = LLM(model="centml/DeepSeek-R1-NVFP4-v2-mlpinf", tensor_parallel_size=8, enable_attention_dp=True)

    outputs = llm.generate(prompts, sampling_params)

    # Print the outputs.
    for output in outputs:
        prompt = output.prompt
        generated_text = output.outputs[0].text
        print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")


# The entry point of the program need to be protected for spawning processes.
if __name__ == '__main__':
    main()