Downloads · 30 days
15
6% of all-time downloads
pygmalion-6b-chaicomp
alkahestry/pygmalion-6b-chaicomp
pygmalion-6b-chaicomp is a text generation model from alkahestry. Use it when you need the model to write or continue text. It is set up for transformers.
All-time downloads
261
Public
Repo size
28.6 GB
Likes
2
Public
Repo makeup
Click a slice to open those files.
.bin16.4 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- Model type
- gptj
- Access
- Public
- Created
- Jun 2, 2023
- Updated
- Jul 26, 2023
- SHA
- 9527e8c2
Try a prompt
Model card
From the Hugging Face model README
Model Details
I finetuned PygmalionAI/pygmalion-6b with QLora for 24 hours on 250k samples. Collected from SODA and Teacher GPT dataset. My first attempt on making LLM model as an entry to Chai competition.
Model Description
- Model type: Chatbot
- Finetuned from model : PygmalionAI/pygmalion-6b
Model Sources
Pygmalion-6b: https://huggingface.co/PygmalionAI/pygmalion-6b
Training Details
Training Data
For the training data I use 20% of SODA dadtaset mixed with TeacherGPT roleplay dataset.
Training Procedure
The model was trained for 24 hours on RTX4090.
Training Hyperparameters
- Training param
batch_size = 128,
micro_batch_size = 4,
num_epochs = 1,
learning_rate = 3e-4,
cutoff_len = 512,
val_set_size = 0
- finetune method
finetune_method = "qlora"
- prefix tuning hyperparams
num_virtual_tokens = 32
- lora hyperparams
lora_r = 16,
lora_alpha = 16,
lora_dropout = 0.05,
lora_target_modules = "q_proj k_proj v_proj"
- llm hyperparams
bf16 = False,
load_in_8bit = False,
group_by_length = False ,
resume_from_checkpoint = None
Results
Me: Hi Nathan, how are you doing today
Nathan: I'm fine...
Me: Then tell me about your day.
Nathan:
It was good. We had a lot of fun in school and then we went to the park afterwards.
- Task
- Text Generation
- Library
- transformers
- Type
- gptj
- Created
- Jun 2, 2023
- Updated
- Jul 26, 2023