DeepSeek-R1-Distill-Qwen-1.5B-GPTQ

DeepSeek-R1-Distill-Qwen-1.5B — GPTQ 4-bit, calibrated and benchmarked by QuantThink

A real, self-calibrated 4-bit GPTQ quantization of deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B, produced and measured by QuantThink — a reproducible benchmark measuring how quantization affects small reasoning (long chain-of-thought) models on a 4GB consumer GPU (RTX 3050 Laptop).

Calibration recipe (exactly what was run)

Reproduce:

from gptqmodel import GPTQModel, QuantizeConfig
from datasets import load_dataset

ds = load_dataset("Salesforce/wikitext", "wikitext-2-raw-v1", split="train")
texts = [r["text"].strip() for r in ds if len(r["text"].strip()) >= 512][:128]

quant_config = QuantizeConfig(bits=4, group_size=128)
model = GPTQModel.load("deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B", quant_config)
model.quantize(calibration=texts, batch_size=1)
model.save("./out-gptq-4bit")

Measured results (RTX 3050 Laptop, 4GB VRAM)

Real evaluation against this project's frozen GSM8K subset (N=6 problems x 2 seeds = 12 samples — same disclosed small-N first-pass discipline as every other result in this project; see docs/RUN_REAL.md):

Loaded and evaluated via gptqmodel's own BACKEND.TORCH inference path (pure PyTorch — the Marlin/Triton kernels need nvcc to JIT-compile, which this driver-only machine doesn't have; see the reproduce command below). Note the VRAM figure here is measured via torch.cuda.max_memory_allocated(), not the nvidia-smi-based measurement used for the GGUF quants elsewhere in this project — the two aren't directly comparable without accounting for that methodology difference.

For context, this project's llama.cpp Q4_K_M GGUF quant of the same base model measured 0.583 Acc / 420.8 TL / 1532.1 CTS at the same sample size (see happynood/DeepSeek-R1-Distill-Qwen-1.5B-GGUF) — this GPTQ quant nominally scores higher on this small sample, plausibly because GPTQ uses real calibration data to minimize reconstruction error while GGUF's Q4_K_M is a calibration-free (round-to-nearest-style) scheme. This is a single N=12 comparison, not a statistically confirmed result — no bootstrap CI has been computed at this sample size.

Reproduce the evaluation

git clone https://github.com/Happynood/quant-reasoning-bench
cd quant-reasoning-bench
uv sync --extra datasets  # for data/suite/gsm8k_e1.jsonl if not already present
pip install gptqmodel torch transformers accelerate ninja
python -c "
from gptqmodel import GPTQModel
from gptqmodel.utils.backend import BACKEND
model = GPTQModel.load('happynood/DeepSeek-R1-Distill-Qwen-1.5B-GPTQ', backend=BACKEND.TORCH)
# ... generate against data/suite/gsm8k_e1.jsonl using quantthink.eval.{extractor,checkers}
"

Related

• GitHub repository
• quantthink-results
• happynood/DeepSeek-R1-Distill-Qwen-1.5B-GGUF — the llama.cpp GGUF quants of the same base model, also benchmarked by this project
• Original model: deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B