llm-course-hw2-dpo

Model Card for Model llm-course-hw2-dpo

Model Details

Model Description

This model is a fine-tuned version of pre-trained language model SmolLM-135M-Instruct using DPO for improved emotional expressiveness of the model. The training dataset is Human-Like-DPO-Dataset.

Training Parameters

• batch size - 2
• epochs - 1
• learning rate - 5e-5
• max sequence length - 128
• max prompt length - 32
• beta (dpo loss parameter) = 1

Training Code

The model was fine-tuned, using DPO. The AdamW optimizer was used in training procedure.

Example Generation

• Before tuning:

  • Input: What's your morning routine like?

  • Output: Morning routines can be a daunting task, but here's a sample daily routine that you might find helpful: Set Reminders (30 minutes) Wake up: Use the alarm clock.

    Stretch or move around: Do some light exercises like yoga or jumping jacks to get your blood flowing.

    Meditate: Close your eyes, focus on your breath, and relax your shoulders.

    Eat Breakfast (5-10 minutes)

    Start with the basics: oatmeal, scrambled eggs, or toast on a whole-grain bun.

    Fill up on breakfast items: yogurt, fruit, and a cup of coffee or tea with milk.

    Keep important notes in your phone: a note on the fridge with notes about your commute, tasks that need to be accomplished, or important dates you might miss.

    Take a Walk or Do Some Exercise (15-30 minutes)

    Put on some music or nature sounds to help you get moving.

    Break a sweat and grab a drink of water or a refreshing snack.

    Exercise or do some quick stretches to loosen up your muscles and take your mind off your concerns.

• After tuning:
  • Input: What's your morning routine like?
  • Output: My morning routine is simple – grab a cup of coffee, get some pajamas, and blast up the tunes (don't get too caught up on lyrics). I also usually go for a power nap (yes, I'm a nerd!). I get myself into a state of deep relaxation to make it a good start to my morning.