Pythia410m-V0-Instruct

Model info

This is EleutherAI/pythia-410m finetuned on OpenAssistant/oasst_top1_2023-08-25

Why

Plain and simple. Im experimenting with making instruction LLMs under 1B params. I think we can still squeeze out better performance out of these models.

Random Notes

• Only using OpenAssistant data gives fantastic results becuase of its high quality. I like the top1 dataset becuase of it's lack of prompt refusals.
• Prompt refusals have been shown to damage the performance of instruction LLMs. My theory is that the model "spends" parameters learning how to refuse prompts rather than learning actually useful information. Adding to this, I think that unlike other tasks, learning prompt refusals most likely has no other value in terms of transfer learning.

Usage

from transformers import pipeline

pipe = pipeline("text-generation", model="SummerSigh/Pythia410m-V0-Instruct")

out= pipe("<|im_start|>user\nWhat's the meaning of life?<|im_end|>\n<|im_start|>assistant\n",max_length  = 500,repetition_penalty = 1.2, temperature  = 0.5, do_sample = True)

print(out[0]["generated_text"])

Contact

If you want to contact me and work with me on making good under 1B param models, you can reach me on discord at summer_ai.