llm-course-hw2-dpo

Model Overview

• Model Type: Causal Language Model fine-tuned with Direct Preference Optimization (DPO)
• Architecture: Based on the HuggingFaceTB/SmolLM-135M-Instruct model
• Summary:
  This model has been fine-tuned using the DPO method, which leverages human preference data (consisting of prompts, a chosen response, and a rejected response) to align the model's outputs with human judgments. Unlike methods that require an explicit reward model, DPO directly optimizes the model based on the log-probability differences between the preferred and non-preferred responses.

Intended Use

• Use Cases:
  • Conversational response generation
  • Question-answering systems
  • Virtual assistant applications where human-preferred outputs are desired
• Limitations:
  • The model is relatively small and might struggle with very complex dialogues or specialized topics.
  • As it is trained on preference data, the model may exhibit biases toward certain stylistic or content-related preferences present in the training set.

Data

• Training Data:
  HumanLLMs/Human-Like-DPO-Dataset – a dataset containing human-annotated preferences.
• Data Preprocessing:
  • The dataset is converted into a chat-template format: the prompt is set as the user message, and the responses are set as the assistant message.
  • The texts are tokenized and truncated appropriately (the prompt is truncated from the left to preserve the most important context).

Training Details

• Optimizer: AdamW
• Learning Rate: 5e-5
• Batch Size: 4 (adjustable based on available resources)
• Training Duration: 1 epoch (this may be adjusted based on resource availability and performance requirements)
• Other Notes:
  Dropout is disabled (all Dropout modules are set to 0) during training to ensure stable inference.

Evaluation and Results

• Generation Comparison:
  The model's output is compared before and after fine-tuning on a sample prompt (e.g., “What's your morning routine like?”) to observe changes in response style and tone.
• Metrics:
  DPO loss, reward accuracy, and reward margins are used to monitor the training process.

Limitations and Ethical Considerations

• Due to the model’s small size and the biases in the training data, the generated responses may sometimes lack depth or present undesired stylistic tendencies.
• It is recommended to thoroughly test the model in sensitive applications and consider additional safeguards if deployed in production environments.

Usage Instructions

• Inference:
  The model is available on HuggingFace Hub and can be accessed via the provided interface.
• Feedback and Updates:
  Users are encouraged to provide feedback to help further improve the model.