Downloads · 30 days
15
4% of all-time downloads
jondurbin/airoboros-mpt-30b-gpt4-1p4-three-epochs
airoboros-mpt-30b-gpt4-1p4-three-epochs is a text generation model from jondurbin. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as other.
This is a test of qlora fine-tuning of the mpt-30b model, with 3 epochs.
Downloads · 30 days
15
4% of all-time downloads
All-time downloads
423
Public
Repo size
120 GB
Likes
0
Public
Click a slice to open those files.
.bin59.9 GB · 100%
From the Hugging Face model README
This is a test of qlora fine-tuning of the mpt-30b model, with 3 epochs.
qlora compatible model: https://huggingface.co/jondurbin/mpt-30b-qlora-compatible
My fork of qlora with mpt-30b support: https://github.com/jondurbin/qlora
Differences in the qlora scripts:
--mpt True for mpt-based models--num_train_epochs instead of --max_stepsI think there's a bug in gradient accumulation, so if you try this, maybe set gradient accumulation steps to 1
See the mpt-30b-qlora-compatible model card for training details.
This is not as high quality as the llama-33b versions unfortunately, but I don't have a great answer as to why. Perhaps there are fewer forward layers that can be tuned?
This is a real gray area, here's why:
I am purposingly not placing a license on here because I am not a lawyer and refuse to attempt to interpret all of the terms accordingly. Your best bet is probably to avoid using this commercially, especially since it didn't perform quite as well as expected using qlora.