Downloads · 30 days
0
RusEnQA
Shavrina/RusEnQA
RusEnQA is a text generation model from Shavrina. Use it when you need the model to write or continue text.
QA for Russian and English based on the rugpt3xl model
Access
Public
Updated Nov 18, 2021
Repo size
2.6 GB
Likes
1
Public
Repo makeup
Click a slice to open those files.
.pt2.6 GB · 100%
At a glance
- Task
- Text Generation
- Access
- Public
- Created
- Mar 2, 2022
- Updated
- Nov 18, 2021
- SHA
- 4f3582e6
Model card
From the Hugging Face model README
RusEnQA
QA for Russian and English based on the rugpt3xl model
Fine-tuning format:
"<s>paragraph: "+eng_context+"\nlang: rus\nquestion: "+rus_question+' answer: '+ rus_answer+"</s>"
About ruGPT-3 XL model
Model was trained with 512 sequence length using Deepspeed and Megatron code by SberDevices team, on 80B tokens dataset for 4 epochs. After that model was finetuned 1 epoch with sequence length 2048.
Note! Model has sparse attention blocks.
Total training time was around 10 days on 256 GPUs. Final perplexity on test set is 12.05. Model parameters: 1.3B.
- Task
- Text Generation
- Languages
- ru, en
- Created
- Mar 2, 2022
- Updated
- Nov 18, 2021