deepseek-r1-dpo

Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->

Better tuned deepseek-r1 model using dpo and specific customer service dataset

Model Details

Model Description

<!-- Provide a longer summary of what this model is. -->

This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.

• Developed by: [More Information Needed]
• Funded by [optional]: [More Information Needed]
• Shared by [optional]: [More Information Needed]
• Model type: [More Information Needed]
• Language(s) (NLP): [More Information Needed]
• License: [More Information Needed]
• Finetuned from model [optional]: [More Information Needed]

Model Sources [optional]

<!-- Provide the basic links for the model. -->

• Repository: [More Information Needed]
• Paper [optional]: [More Information Needed]
• Demo [optional]: [More Information Needed]

How to use this model

first we need unsloth

Normally using pip install unsloth is enough

Temporarily as of Jan 31st 2025, Colab has some issues with Pytorch

Using pip install unsloth will take 3 minutes, whilst the below takes <1 minute:

%%capture !pip install --no-deps bitsandbytes accelerate xformers==0.0.29 peft trl triton !pip install --no-deps cut_cross_entropy unsloth_zoo !pip install sentencepiece protobuf datasets huggingface_hub hf_transfer !pip install --no-deps unsloth

from unsloth import FastLanguageModel from transformers import AutoTokenizer

Path to your fine-tuned model

model_path = "drive/MyDrive/deepseek-r1-reasoning-dpo" # Replace

Load the base model optimized with Unsloth

model, tokenizer = FastLanguageModel.from_pretrained( model_name=model_path, max_seq_length=4096, # Adjust based on model capability dtype=torch.float16, load_in_4bit=True, # Enable quantization for efficiency )

Optimize LoRA model for inference (2x faster with Unsloth)

FastLanguageModel.for_inference(model)

Move model to GPU if available

device = "cuda" if torch.cuda.is_available() else "cpu" model.to(device)

print("Model loaded successfully!")

---------------------------------------------