DeepSeek-R1-Distill-Qwen-1.5B-GGUF

DeepSeek-R1-Distill-Qwen-1.5B — GGUF, benchmarked by QuantThink

GGUF quantizations of deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B, originally converted by bartowski (re-hosted here unmodified — same files, same quantization recipe, all credit for the GGUF conversion itself belongs to bartowski). What's new here is the benchmark data: real, measured accuracy/thinking-length/cost-to-solve for each quant level from QuantThink, a reproducible benchmark measuring how quantization affects small reasoning (long chain-of-thought) models on a 4GB consumer GPU.

Measured results (RTX 3050 Laptop, 4GB VRAM)

First-pass numbers (N=6 problems x 2 seeds = 12 samples per cell — disclosed small-N first pass, not yet the full statistically-rigorous sweep; see docs/RUN_REAL.md for the complete write-up and caveats):

Q4_K_M is the only quant showing a measurable accuracy drop vs. fp16 (-8.3 points on both benchmarks) at this sample size; Q8_0 and Q5_K_M show no measurable degradation. See the GitHub repo's docs/RUN_REAL.md for KV-cache-quantization results (including a genuine finding: Q4 KV cache causes total generation collapse for this model, not smooth degradation), thinking-cap results, and the Memory-Budget Frontier this data feeds.

Sampling used for these numbers

Temperature 0.6, top-p 0.95 (per the model card's own recommendation), fixed seed set [0, 1] — not greedy decoding, since reasoning models are known to degrade under greedy sampling.

Reproduce

git clone https://github.com/Happynood/quant-reasoning-bench
cd quant-reasoning-bench
uv sync --extra llama-cpp
uv run quantthink run --config configs/phase1/Q4_K_M_E1.yaml --output result.json --manifest manifest.json

Related

• GitHub repository
• quantthink-results — all real result.json files
• quantthink-leaderboard — interactive leaderboard
• Original GGUF conversion: bartowski/DeepSeek-R1-Distill-Qwen-1.5B-GGUF
• Original model: deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B