DeepSeek-DQ3_K_M

Quantitative Analysis of Performance Drop in DeepSeek Model Quantization

Enbo Zhao^1,2, Yi Shen^1,2, Shuming Shi^1,2, Jieyun Huang^1,2, Zhihao Chen^1,2, Ning Wang^1,2, Siqi Xiao^1,2, Jian Zhang^1,2, Kai Wang^1,2, Shiguo Lian^1,2

¹ Unicom Data Intelligence, China Unicom
² Data Science & Artificial Intelligence Research Institute, China Unicom

Paper Link

• Arxiv: Quantitative Analysis of Performance Drop in DeepSeek Model Quantization

Abstract

Recently, there is a high demand for deploying DeepSeek-R1 and V3 locally, possibly because the official service often suffers from being busy and some organizations have data privacy concerns. While single-machine deployment offers infrastructure simplicity, the models’ 671B FP8 parameter configuration exceeds the practical memory limits of standard 8-GPU devices (A100/H100/910B). Quantization is a widely used technique that helps reduce model memory consumption. However, it is unclear what the performance of DeepSeek-R1 and V3 will be after being quantized. This technical report presents the first comprehensive evaluation of multibitwidth quantization across the complete DeepSeek model spectrum. Key findings reveal that 4-bit quantization maintains little performance degradation versus FP8 while enabling single-machine deployment on standard Nvidia GPU devices. We further propose DQ3_K_M, a dynamic 3-bit quantization method that significantly outperforms traditional Q3_K_M variantion various benchmarks, which is also comparable with 4-bit quantization (Q4_K_M) approach in most tasks. Moreover, DQ3_K_M supports single-machine deployment configurations for both NVIDIA H100/A100 and Huawei 910B.

Experimental Results

Table 1: Resource usage of DQ3_K_M versus llama.cpp and Unsloth quantizations for DeepSeek R1 (671B) at a 32K‑token context length.

<br>

Table 2: Quantization results of DeepSeek‑R1 on various benchmarks.

<br>

Table 3: Quantization results of DeepSeek-V3 on various benchmarks.

<br>

Table 4: Quantization results of DeepSeek-R1-distill-Qwen-32B on various benchmarks