Downloads · 30 days
9
26% of all-time downloads
llama3-dpo-pairrm
pyamy/llama3-dpo-pairrm
llama3-dpo-pairrm is a machine learning model from pyamy. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for peft. The card lists the license as apache-2.0.
This model is a fine-tuned version of meta-llama/Llama-3.2-1B-Instruct using Direct Preference Optimization (DPO).
All-time downloads
34
Public
Repo size
336 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json121 MB · 50%
At a glance
- Library
- peft
- License
- apache-2.0
- Access
- Public
- Created
- Aug 11, 2025
- Updated
- Aug 12, 2025
- SHA
- 8f537e62
Base models
Model card
From the Hugging Face model README
Llama-3.2-1B DPO PairRM
This model is a fine-tuned version of meta-llama/Llama-3.2-1B-Instruct using Direct Preference Optimization (DPO).
Model Details
- Base Model: meta-llama/Llama-3.2-1B-Instruct
- Training Method: Direct Preference Optimization (DPO)
- Preference Source: PairRM
- LoRA Configuration:
- r: 8
- alpha: 16
- target_modules: ['q_proj', 'k_proj', 'v_proj', 'o_proj']
- Training Steps: 250
- Learning Rate: 0.0002
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B-Instruct")
model = PeftModel.from_pretrained(base_model, "pyamy/llama3-dpo-pairrm")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B-Instruct")
Training Details
- Dataset: 50 instructions from LIMA
- Responses per instruction: 5
- Preference judgment: PairRM
- Training framework: TRL DPOTrainer
Performance
See evaluation results in the repository for detailed performance metrics.
- Library
- peft
- License
- apache-2.0
- Created
- Aug 11, 2025
- Updated
- Aug 12, 2025