Downloads · 30 days
31
28% of all-time downloads
DeepSeek-R1-Distill-Qwen-7B-GPTQ-Int8
frankdarkluo/DeepSeek-R1-Distill-Qwen-7B-GPTQ-Int8
DeepSeek-R1-Distill-Qwen-7B-GPTQ-Int8 is a text generation model from frankdarkluo. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as mit.
See details on the official page of the model: DeepSeek-R1-Distill-Qwen-7B
All-time downloads
112
Public
Parameters
7.6B
8.9 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors8.9 GB · 100%
Parameter types
How the weights are stored.
I326.5B · 86%
Try a prompt
Base models
Model card
From the Hugging Face model README
GPTQ 8bit quantized version of DeepSeek-R1-Distill-Qwen-7B
Model Details
See details on the official page of the model: DeepSeek-R1-Distill-Qwen-7B
Quantized using GPTQModel using Allenai/C4 dataset. Quantization config:
bits=8,
group_size=128,
desc_act=False,
How to use
Using transformers library with integrated GPTQ support:
from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig
model_name = "frankdarkluo/DeepSeek-R1-Distill-Qwen-7B-GPTQ-Int8"
tokenizer = AutoTokenizer.from_pretrained(model_name)
quantized_model = AutoModelForCausalLM.from_pretrained(model_name, device_map='cuda')
chat = [{"role": "user", "content": "Why is grass green?"},]
question_tokens = tokenizer.apply_chat_template(chat, add_generation_prompt=True, return_tensors="pt").to(quantized_model.device)
answer_tokens = quantized_model.generate(question_tokens, generation_config=GenerationConfig(max_length=2048, ))[0]
print(tokenizer.decode(answer_tokens))
- Task
- Text Generation
- Library
- transformers
- Type
- qwen2
- License
- mit
- Created
- Dec 13, 2025
- Updated
- Dec 26, 2025