Downloads · 30 days
0
rugpt3xl
ai-forever/rugpt3xl
rugpt3xl is a machine learning model from ai-forever. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Model was trained with 512 sequence length using Deepspeed and Megatron code by SberDevices team, on 80B tokens dataset for 4 epochs. After that model was finetuned 1 epoch with sequence length 2048. Note! Model has s…
Access
Public
Updated Sep 21, 2021
Repo size
2.6 GB
Likes
19
Public
Repo makeup
Click a slice to open those files.
.pt2.6 GB · 100%
At a glance
- Access
- Public
- Created
- Mar 2, 2022
- Updated
- Sep 21, 2021
- SHA
- c3c28881
Model card
From the Hugging Face model README
rugpt3xl
Model was trained with 512 sequence length using Deepspeed and Megatron code by SberDevices team, on 80B tokens dataset for 4 epochs. After that model was finetuned 1 epoch with sequence length 2048.
Note! Model has sparse attention blocks.
Total training time was around 10 days on 256 GPUs.
Final perplexity on test set is 12.05.
Model parameters: 1.3B.
- Languages
- ru
- Created
- Mar 2, 2022
- Updated
- Sep 21, 2021