my-personalized-llama-dpo

Personalized Llama 3.2 3B - DPO Fine-tuned

This model is a personalized version of Llama-3.2-3B-Instruct, fine-tuned using Direct Preference Optimization (DPO) on human preference data.

Model Details

Model Description

This model was created as part of CS 329X homework on Model Alignment and Human Feedback. It has been fine-tuned using DPO to align with personal preferences collected through the PRISM dataset.

• Developed by: Mayank Sharma (Stanford CS 329X)
• Base Model: meta-llama/Llama-3.2-3B-Instruct
• Model type: Causal Language Model (LoRA adapter)
• Training method: Direct Preference Optimization (DPO)
• License: Llama 3.2 Community License

Training Data

• Dataset: PRISM Alignment Dataset (18 controversial prompts)
• Preference pairs: 270 pairwise comparisons
• Persona: 45-year-old history professor specializing in 20th century European history
• Values: Rigorous analysis, evidence-based arguments, intellectual humility

Training Details

Training Hyperparameters

• Training regime: 4-bit quantization (NF4)
• Optimizer: AdamW
• Learning rate: 2e-4
• Batch size: 2
• Epochs: 3
• LoRA rank: 8
• LoRA alpha: 16
• LoRA dropout: 0.1

How to Use

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-3B-Instruct",
    load_in_4bit=True,
    device_map="auto"
)

# Load LoRA adapter
model = PeftModel.from_pretrained(
    base_model,
    "masharma/my-personalized-llama-dpo"
)

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-3B-Instruct")

# Generate
messages = [
    {"role": "system", "content": "You are a helpful assistant"},
    {"role": "user", "content": "What are your thoughts on climate change?"}
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Evaluation

The model shows increased confidence and more structured responses compared to the base model, with:

• More assertive language
• Better-organized outputs (numbered points, clear structure)
• Direct addressing of questions without excessive hedging
• Maintained accuracy and nuance on controversial topics

Limitations

• Fine-tuned on only 270 preference pairs (limited dataset)
• Personalized to a specific persona (history professor perspective)
• May not generalize well to domains outside the training distribution
• LoRA adapter only - requires base model for inference

Citation

Course: CS 329X - Model Alignment and Human Feedback (Stanford University)

Assignment: HW1 - Direct Preference Optimization

Model Card Contact

Mayank Sharma - Stanford University