pygmalion_6b_roleplay_lora

Trained in 4-bit on pygmalion-6b as POC

Uses the GPTeacher roleplay dataset.

INFO:Getting model ready...                                                                                                        
INFO:Prepping for training...
INFO:Creating LoRA model...
INFO:Starting training...
{'loss': 12.5737, 'learning_rate': 0.0002926829268292683, 'epoch': 0.33}
{'loss': 8.5515, 'learning_rate': 0.0002560975609756097, 'epoch': 0.67}
{'loss': 7.5768, 'learning_rate': 0.0002195121951219512, 'epoch': 1.0}
{'loss': 6.9769, 'learning_rate': 0.00018292682926829266, 'epoch': 1.33}
{'loss': 6.6842, 'learning_rate': 0.00014634146341463414, 'epoch': 1.66}
{'loss': 6.3925, 'learning_rate': 0.0001097560975609756, 'epoch': 2.0}
{'loss': 6.041, 'learning_rate': 7.317073170731707e-05, 'epoch': 2.33}
{'loss': 5.6818, 'learning_rate': 3.6585365853658535e-05, 'epoch': 2.66}
{'loss': 5.4639, 'learning_rate': 0.0, 'epoch': 2.99}
{'train_runtime': 960.7748, 'train_samples_per_second': 6.005, 'train_steps_per_second': 0.047, 'train_loss': 7.326934729682074, 'epoch': 2.99}
INFO:LoRA training run is completed and saved.
INFO:Training complete!

I used the electricity so might as well post it.