llm-course-hw2-dpo

Модель, выровненная методом DPO

Реализована ручками с нуля, включая dpo loss

Гиперпараметры обучения

BATCH_SIZE = 4 NUM_EPOCHS = 3 LR = 5e-6 MAX_SEQ_LEN = 1024 MAX_PROMPT_LEN = 256 MAX_COMPLETION_LEN = None BETA = 0.1

Результаты обучения (accuracy и margins)

Accuracy --> 1 Margins --> 100

Пример генерации и сравнение с генерацией до выравнивания:

======== BEFORE TUNING ======== user What's your morning routine like? assistant What a wonderful question! As a conversational AI, I have my own wake-up routine tailored to my needs. Here's my "Awake Morning Routine" for you:

Morning Routine (10-15 minutes)

1. 10:00 AM: Start with a gentle exercise routine to loosen up my muscles and get my blood flowing. This could be a simple yoga pose like downward-facing dog, child's pose, or a standing forward fold.
2. 11:00 AM: Engage in a gentle stretching exercise to loosen up my muscles and prepare for the next day. This can include chest opens, shoulder rolls, or neck stretches. Hold each stretch for 20-30 seconds and then relaxation.
3. 5:00 AM: Drink a caffeine-free warm or cold drink to start my day on a positive note. This could be a joe-toap, herbal tea, or a cold beverage like peppermint or ginger.
4. 7:00 AM: Take a mid-morning coffee shop break or grab a quick snack from a grab-and-go supply like a whole wheat toast or a bag of chips. This will give my body

======== AFTER TUNING ======== user What's your morning routine like? assistant Hey! Do you have a morning routine that works for you? Different people have different schedules, but I figured I'd share my top tips:

** wake up**

• Start your day by doing some morning yoga or stretching to set a positive tone for the morning.
• Take a 5-10 minute power nap to refresh your energy and refresh your mind.

** Breakfast**

• Just grab a quick breakfast (e.g., oatmeal with banana, almond butter) and don't worry about it. You can always grab a pre-washed fridge for lunch.
• Try these:
  • Avocado toast with scrambled eggs and a sprinkle of salt and pepper
  • Whole-grain toast with peanut butter and banana
  • Smoothie bowl with banana, almond milk, protein powder, and chia seeds

** Exercise routine (optional)**

• If you're looking for a quick and low-intensity workout, start with a 10-15 minute walk or some light stretching.
• You can also try some yoga or stretching exercises to get some morning movements on your nerves.

** Get ready for the day (if you can)**

• Take a shower or