Downloads · 30 days
17
3% of all-time downloads
byt5-dv
monsoon-nlp/byt5-dv
byt5-dv is a machine learning model from monsoon-nlp. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers.
Pretrained from scratch on Dhivei (language of the Maldives) with ByT5, Google's new byte-level tokenizer strategy.
All-time downloads
623
Public
Parameters
300M
3.6 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.h51.2 GB · 33%
At a glance
- Library
- transformers
- Model type
- t5
- Access
- Public
- Created
- Mar 2, 2022
- Updated
- Feb 24, 2025
- SHA
- 7a4495bb
Model card
From the Hugging Face model README
byt5-dv
Pretrained from scratch on Dhivei (language of the Maldives) with ByT5, Google's new byte-level tokenizer strategy.
Corpus: dv.wikipedia.org as of March 2020 (TFDS)
Notebook - Pretraining on Wikipedia: https://colab.research.google.com/drive/19Afq7CI6cOi1DaTpnQhBbEbnBzLSFHbH
Demo
Notebook - Finetuning on Maldivian news classification task: https://colab.research.google.com/drive/11u5SafR4bKICmArgDl6KQ9vqfYtDpyWp
Current performance:
- mBERT: 52%
- byt5-dv: 81%
- dv-wave (ELECTRA): 89%
- dv-muril: 90.7%
- dv-labse: 91.3-91.5%
Source of dataset: https://github.com/Sofwath/DhivehiDatasets
Work in progress - todos
The Wikipedia corpus is too small for this language. In the future I would add OSCAR and Sofwath's Maldivian corpus, if I can rewrite the script to accept those as one TFDS dataset.
This is based on ByT5-small ... we should try a larger model
This needs more time for pretraining
- Library
- transformers
- Type
- t5
- Languages
- dv
- Created
- Mar 2, 2022
- Updated
- Feb 24, 2025