Downloads · 30 days
39
1% of all-time downloads
DeepSeek-R1-Distill-Qwen-32B-GPTQ-Int8
frankdarkluo/DeepSeek-R1-Distill-Qwen-32B-GPTQ-Int8
DeepSeek-R1-Distill-Qwen-32B-GPTQ-Int8 is a text generation model from frankdarkluo. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as mit.
See details on the official page of the model: DeepSeek-R1-Distill-Qwen-32B
All-time downloads
3.7K
Public
Parameters
32.8B
35.1 GB on disk
Likes
2
Public
Repo makeup
Click a slice to open those files.
.safetensors35.1 GB · 100%
Parameter types
How the weights are stored.
I3231.2B · 95%
Try a prompt
Base models
Model card
From the Hugging Face model README
GPTQ 8bit quantized version of DeepSeek-R1-Distill-Qwen-32B
Model Details
See details on the official page of the model: DeepSeek-R1-Distill-Qwen-32B
Quantized using GPTQModel using Allenai/C4 dataset. Quantization config:
bits=8,
group_size=128,
desc_act=False,
How to use
Using transformers library with integrated GPTQ support:
from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig
model_name = "frankdarkluo/DeepSeek-R1-Distill-Qwen-32B-GPTQ-Int8"
tokenizer = AutoTokenizer.from_pretrained(model_name)
quantized_model = AutoModelForCausalLM.from_pretrained(model_name, device_map='cuda')
chat = [{"role": "user", "content": "Why is grass green?"},]
question_tokens = tokenizer.apply_chat_template(chat, add_generation_prompt=True, return_tensors="pt").to(quantized_model.device)
answer_tokens = quantized_model.generate(question_tokens, generation_config=GenerationConfig(max_length=2048, ))[0]
print(tokenizer.decode(answer_tokens))
- Task
- Text Generation
- Library
- transformers
- Type
- qwen2
- License
- mit
- Created
- Dec 13, 2025
- Updated
- Dec 26, 2025