DeepSeek-R1-Distill-Qwen-7B-to-GGUF

DeepSeek-R1-Distill-Qwen-7B GGUF

Community-made GGUF conversions and quantizations of DeepSeek-R1-Distill-Qwen-7B, intended for local inference with GGUF-compatible software.

The original model was converted from its Hugging Face / SafeTensors distribution to GGUF using tools provided by llama.cpp.

This is an unofficial community conversion.

The original DeepSeek-R1-Distill-Qwen-7B model was developed and released by DeepSeek. It is based on Qwen2.5-Math-7B and was fine-tuned using reasoning data generated by DeepSeek-R1. This repository only provides converted and quantized GGUF files derived from the original model.

Original Model

• Model: deepseek-ai/DeepSeek-R1-Distill-Qwen-7B
• Developer: DeepSeek
• Base model: Qwen2.5-Math-7B
• Original format: SafeTensors
• License: MIT License
• Original model: https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B

Refer to the original model repository for the authoritative model card, capabilities, limitations, usage recommendations, license terms, and information about the DeepSeek-R1 distillation process.

Available GGUF Files

This repository provides the original GGUF conversion alongside several quantized variants:

Actual memory consumption may be higher than the GGUF file size and depends on factors such as context length, KV cache configuration, inference backend, GPU offloading, and runtime settings.

Conversion Pipeline

The files in this repository were produced using a workflow based on llama.cpp:

DeepSeek-R1-Distill-Qwen-7B
             │
             │ SafeTensors
             ▼
    convert_hf_to_gguf.py
             │
             ▼
         BF16 GGUF
             │
             │ llama-quantize
             ▼
   ┌────────┬────────┬───────┬──────┐
   │Q4_K_M  │Q5_K_M  │ Q6_K  │ Q8_0 │
   └────────┴────────┴───────┴──────┘

No additional training, fine-tuning, or distillation is performed as part of this conversion process.

The reasoning capabilities and distilled model weights originate from the upstream DeepSeek model. This repository only changes the model representation through GGUF conversion and quantization.

Quantization changes the numerical representation of the model weights to reduce storage and memory requirements and may affect model quality.

Usage

These GGUF files are intended for applications and inference engines with compatible GGUF support, particularly llama.cpp and software built around it.

Example with llama.cpp:

llama-cli \
  -m DeepSeek-R1-Distill-Qwen-7B-Q5_K_M.gguf \
  -p "Solve the equation 3x + 7 = 22 and explain your reasoning."

Runtime parameters should be adjusted according to your hardware, available memory, desired context length, and inference backend.

For model-specific prompting recommendations and expected behavior, refer to the original DeepSeek model card.

Compatibility

GGUF compatibility depends on the version of llama.cpp and its support for the underlying model architecture.

Because both llama.cpp and GGUF continue to evolve, older inference engines may not correctly load files produced by newer versions.

If you encounter GGUF compatibility problems, first test with a recent version of llama.cpp or your preferred GGUF-compatible runtime.

Reproducibility

The conversion process follows the standard Hugging Face / SafeTensors → GGUF workflow provided by llama.cpp.

The general process consists of:

1. obtaining the original deepseek-ai/DeepSeek-R1-Distill-Qwen-7B SafeTensors model;
2. converting the model to GGUF using convert_hf_to_gguf.py;
3. retaining the resulting BF16 GGUF;
4. quantizing the BF16 GGUF using llama-quantize;
5. producing the Q4_K_M, Q5_K_M, Q6_K, and Q8_0 variants.

Conversion and quantization behavior may vary between llama.cpp revisions as model architecture support and GGUF tooling evolve.

Credits

DeepSeek

The original DeepSeek-R1-Distill-Qwen-7B model and its distilled weights were developed and released by DeepSeek.

The model was created by fine-tuning Qwen2.5-Math-7B using reasoning data generated by DeepSeek-R1.

This repository would not exist without DeepSeek's work and release of the original model.

• DeepSeek: https://www.deepseek.com/
• Original model: https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B

All credit for the original distilled model belongs to DeepSeek and its respective authors and contributors.

Qwen / Alibaba

DeepSeek-R1-Distill-Qwen-7B is based on Qwen2.5-Math-7B, developed by the Qwen Team at Alibaba.

Credit for the underlying Qwen architecture and base model belongs to the Qwen Team and its contributors.

• Qwen: https://qwen.ai/
• Qwen on Hugging Face: https://huggingface.co/Qwen

llama.cpp

GGUF conversion and quantization are performed using tools from the open-source llama.cpp project.

This workflow relies on tooling including:

• convert_hf_to_gguf.py
• llama-quantize
• GGUF infrastructure provided by the project

Project:

https://github.com/ggml-org/llama.cpp

Credit belongs to the llama.cpp maintainers and contributors for the conversion, quantization, GGUF, and local inference tooling used by this workflow.

License

The original DeepSeek-R1-Distill-Qwen-7B model is distributed under the MIT License.

According to the upstream DeepSeek-R1 repository, the Qwen-based distilled models are derived from Qwen2.5 series models and are distributed under the same MIT terms specified for the DeepSeek-R1 release.

These files are converted and quantized derivatives of the original DeepSeek-R1-Distill-Qwen-7B model weights and do not introduce additional model training or fine-tuning.

Please review the original DeepSeek-R1-Distill-Qwen-7B model repository and its licensing information before using or redistributing these files.

Disclaimer

This repository is an unofficial community conversion and is not affiliated with, endorsed by, or maintained by DeepSeek, Alibaba, the Qwen Team, Vast.ai, or the llama.cpp project.

Vast.ai was used as the environment in which the conversion workflow was tested. Its use does not imply affiliation, endorsement, or a technical requirement to use Vast.ai.

The purpose of this repository is to provide GGUF variants of the original openly released model for local inference while documenting and crediting the upstream projects used to create them.