llama-3.2-3b-dpo-LLM-as-Judge-preference

Llama-3.2-3B-DPO-LLM-as-Judge-Preference

This model is a fine-tuned version of meta-llama/Llama-3.2-3B-instruct using Direct Preference Optimization (DPO) on a preference dataset created by an LLM judge.

Model Details

• Base Model: meta-llama/Llama-3.2-3B-instruct
• Fine-tuning Method: Direct Preference Optimization (DPO)
• Training Data: LLM judge-generated preference dataset (500 entries)
• PEFT Method: QLoRA (Quantized Low-Rank Adaptation)

Training Parameters

• LoRA Rank: 16
• LoRA Alpha: 32
• Learning Rate: 5e-5
• Training Steps: 200
• Batch Size: 1 (with gradient accumulation of 8)
• DPO Beta: 0.1

Usage

from peft import PeftModel, PeftConfig
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load the model
model_name = "abhisheksagar/llama-3.2-3b-dpo-LLM-as-Judge-preference" 
config = PeftConfig.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    config.base_model_name_or_path,
    torch_dtype=torch.float16,
    device_map="auto"
)
model = PeftModel.from_pretrained(model, model_name)
tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)

# Generate text
input_text = "What's the most important concept in machine learning?"
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=256,
    temperature=0.7,
    top_p=0.9
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))