dpo-llama3-lora-judge

Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->

Model Details

Model Description

<!-- Provide a longer summary of what this model is. -->

DPO Fine-Tuning Report: LLaMA 3.2 + LLM Judge Preference Dataset

Overview

This document outlines the training process, configuration, and usage of the DPO fine-tuned LLaMA 3.2 model using the LLM Judge preference dataset. The fine-tuned model is available on the Hugging Face Hub at: sxsun1684/dpo-llama3-lora-judge.

Model Information

• Base model: meta-llama/Llama-3.2-1B
• Fine-tuning method: Direct Preference Optimization (DPO)
• LoRA adaptation: Enabled with peft
• Preference dataset: sxsun1684/llm_judge_lima50_preferences

Dataset Summary

• Dataset format: LLM-generated preference pairs with prompt, chosen, and rejected fields
• Size: 75 examples
• Origin: Annotated responses based on 50 seed instructions from the LIMA dataset

Preprocessing

• Tokenization:
  • prompt: max length 128
  • chosen/rejected: max length 384
  • All sequences padded to max length using tokenizer.eos_token
• Output fields:
  • prompt_input_ids, prompt_attention_mask
  • chosen_input_ids, chosen_attention_mask
  • rejected_input_ids, rejected_attention_mask

Training Configuration

DPOConfig(
    beta=0.1,
    learning_rate=2e-5,
    per_device_train_batch_size=1,
    gradient_accumulation_steps=8,
    num_train_epochs=3,
    max_length=512,
    save_strategy="epoch",
    logging_steps=10,
    push_to_hub=False,
    report_to="none",
)

PEFT Configuration (LoRA)

LoraConfig(
    r=8,
    lora_alpha=16,
    bias="none",
    task_type="CAUSAL_LM",
)

Training Details

• Total steps: ~27
• Loss range: ~0.64 → 0.60
• Environment: Colab Pro, 15GB GPU

Usage Example

from transformers import AutoTokenizer, AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("sxsun1684/dpo-llama3-lora-judge")
tokenizer = AutoTokenizer.from_pretrained("sxsun1684/dpo-llama3-lora-judge")

prompt = "Why do cats knead blankets before sleeping?"
input_ids = tokenizer(prompt, return_tensors="pt").input_ids
output = model.generate(input_ids, max_new_tokens=100)
print(tokenizer.decode(output[0], skip_special_tokens=True))

License & Credits

• Base model licensed by Meta
• Fine-tuned model for academic, non-commercial use
• Author: sxsun1684
• Dataset credits: Based on work from GAIR and OpenAssistant

---

For questions or updates, contact the model maintainer via Hugging Face or GitHub.