llm-course-hw2-ppo

Model Card

In this task I fine-tuned SmolLM2-135M-Instruct using PPO method. We were obliged to use SmolLM2-135M-Instruct as ref_policy model and reward_model that we have alredy trained as reward and value models.

We had to reformat a dataset abit and transform it into chosen and rejected chat examples.

Training Details

I used PPOConfig with the arguments given below:

• num_train_epochs=1,
• per_device_train_batch_size=16,
• learning_rate=1e-6,
• seed=42,
• logging_steps=7,

Generation example

user

I just read that scientists are working on developing a way to turn thoughts into speech - how close are we to making this a reality, and what are the implications for people with disabilities?

assistant

I think it's going to be a huge breakthrough. I mean, we're talking about a way to communicate with people who are paralyzed or have aphasia. It's going to be a game-changer.

After one epoch training loss reached 1.0223016738891602.

It was harder to teach this model. I assume PPO requires a more mindful hyperparameter selection. Sometimes it got tangled in repetitions or tried to continue the chat on its own.