Downloads · 30 days
83
6% of all-time downloads
ru-rope-t5-small-instruct
melmoth/ru-rope-t5-small-instruct
ru-rope-t5-small-instruct is a machine learning model from melmoth. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as apache-2.0.
The Russian Rotary Position Embedding T5 model of small version after instruct tuning
All-time downloads
1.3K
Public
Parameters
77.1M
308 MB on disk
Likes
21
Public
Repo makeup
Click a slice to open those files.
.safetensors308 MB · 100%
At a glance
- Library
- transformers
- License
- apache-2.0
- Model type
- t5
- Access
- Public
- Created
- May 30, 2024
- Updated
- Jun 1, 2024
- SHA
- e7148b2a
Model card
From the Hugging Face model README
Model Card for ru-rope-t5-small-instruct
The Russian Rotary Position Embedding T5 model of small version after instruct tuning
Model Details
The model was trained in a Russian corpus with a mix of English using the Mixture-Of-Denoisers pre-training method by UL2 on 1024 length sequences. Training using Flash Attention 2 is available because of the replacement of bias with rotary encoding.
- Model type: RoPE T5
- Language(s) (NLP): Russian, English
Uses
Finetuning for downstream tasks
Bias, Risks, and Limitations
Despite the instructional tuning, it is not recommended to use in zero-shot mode due to the small size
Training Details
Training Data
A corpus of Russian texts from Vikhr filtered by FRED-T5-1.7B perplexy. Instructions are translated English set
Training Procedure
Using AdamWScale instead of Adafactor for stable learning without loss explosions
Metrics
Model Card Contact
- Library
- transformers
- Type
- t5
- License
- apache-2.0
- Languages
- ru, en
- Created
- May 30, 2024
- Updated
- Jun 1, 2024