dpo-llama3-lora-pairrm

Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->

DPO Fine-Tuning Report: LLaMA 3.2 + PairRM Preference Dataset

Model: sxsun1684/dpo-llama3-lora-pairrm

1. Overview

This document summarizes the process and configuration used to fine-tune the LLaMA 3.2 1B model using the PairRM preference dataset through Direct Preference Optimization (DPO) and PEFT (LoRA).

---

2. Objective

To improve the model's alignment with human preferences by fine-tuning it on pairwise preference data (chosen vs rejected responses) using DPO, leveraging PairRM-generated labels.

---

3. Dataset

• Name: sxsun1684/pairrm-lima50-preferences
• Size: 75 instruction pairs
• Format: Each example contains:
  • prompt: the instruction/query
  • chosen: the preferred response
  • rejected: the less preferred response

---

4. Model Setup

• Base Model: meta-llama/Llama-3.2-1B
• PEFT Method: LoRA (Low-Rank Adaptation)

LoRA Configuration

LoraConfig(
    r=8,
    lora_alpha=16,
    bias="none",
    task_type="CAUSAL_LM"
)

---

5. DPO Training Configuration

DPOConfig(
    beta=0.1,
    learning_rate=2e-5,
    per_device_train_batch_size=1,
    gradient_accumulation_steps=8,  # effective batch size = 8
    num_train_epochs=3,
    max_length=512,
    save_strategy="epoch",
    logging_steps=10,
    push_to_hub=False,
    report_to="none",
    padding_value=tokenizer.pad_token_id
)

---

6. Preprocessing

• Each of prompt, chosen, and rejected was tokenized separately.
• Max lengths:
  • Prompt: 128 tokens
  • Chosen & Rejected: 384 tokens
• Padding: max_length with EOS as pad token

---

7. Training Notes

• Used DPOTrainer from trl==0.16.1
• No evaluation dataset (only training)
• Training completed in ~3 epochs without OOM errors on batch size 1

---

8. Output

• Model saved to: sxsun1684/dpo-llama3-lora-pairrm
• Contains fine-tuned LoRA adapters and tokenizer config

---

9. Suggested Use

You can load and use the model with:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("sxsun1684/dpo-llama3-lora-pairrm")
tokenizer = AutoTokenizer.from_pretrained("sxsun1684/dpo-llama3-lora-pairrm")

---

10. Next Steps

• Compare completions on novel instructions vs base LLaMA and LLM Judge DPO model
• Run qualitative/quantitative analysis of improvements
• Optionally deploy via Gradio or Hugging Face Spaces

---

Author

SX Sun (sxsun1684)
2025-04