chat-opt-1.3b

LoRA for OPT-1.3b, finetuned for chatting. Note that this model is meant to be conversation tuned, NOT instruct tuned.

Trained on a selection of data from https://www.kaggle.com/datasets/jef1056/discord-data

Prompting format:

[Person 1]: [message]
[Person 2]: [message]
[Person 1]:

Example chat:

human: hello
bot: hi how are you?
human: good, you?
bot: I'm doing alright. How about yourself?
human: what are you doing now?
bot: just watching netflix and playing games. What about you?
human: nothing much
bot: same here
human: write me a poem
bot: ok i will do it when im not on my phone

Training params:

transformers.TrainingArguments(
    per_device_train_batch_size=4, 
    fp16=True,
    warmup_steps=30,
    max_steps=200,
    learning_rate=1.7e-4,
    logging_steps=1, 
    output_dir='outputs',
    report_to="none"
)