Downloads · 30 days
13
2% of all-time downloads
dv-muril
monsoon-nlp/dv-muril
dv-muril is a fill-mask model from monsoon-nlp. Use it when you need the model to fill a missing word. It is set up for transformers.
This is an experiment in transfer learning, to insert Dhivehi word and word-piece tokens into Google's MuRIL model.
All-time downloads
776
Public
Parameters
245M
5 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.bin979 MB · 33%
Parameter types
How the weights are stored.
F32245M · 100%
Try a prompt
Model card
From the Hugging Face model README
dv-muril
This is an experiment in transfer learning, to insert Dhivehi word and word-piece tokens into Google's MuRIL model.
This BERT-based model currently performs better than dv-wave ELECTRA on the Maldivian News Classification task https://github.com/Sofwath/DhivehiDatasets
Training
- Start with MuRIL (similar to mBERT) with no Thaana vocabulary
- Based on PanLex dictionaries, attach 1,100 Dhivehi words to Malayalam or English embeddings
- Add remaining words and word-pieces from BertWordPieceTokenizer / vocab.txt
- Continue BERT pretraining
Performance
- mBERT: 52%
- dv-wave (ELECTRA, 30k vocab): 89%
- dv-muril (10k vocab) before BERT pretraining step: 89.8%
- previous dv-muril (30k vocab): 90.7%
- dv-muril (10k vocab): 91.6%
CoLab notebook: https://colab.research.google.com/drive/113o6vkLZRkm6OwhTHrvE0x6QPpavj0fn
- Task
- Fill-Mask
- Library
- transformers
- Type
- bert
- Languages
- dv
- Created
- Mar 2, 2022
- Updated
- Mar 10, 2025