Downloads · 30 days
69
1% of all-time downloads
KanBERTo
Naveen-k/KanBERTo
KanBERTo is a fill-mask model from Naveen-k. Use it when you need the model to fill a missing word. It is set up for transformers.
This is a small language model for Kannada language with 1M data samples taken from OSCAR page
All-time downloads
8.9K
Public
Repo size
1.3 GB
Likes
1
Public
Repo makeup
Click a slice to open those files.
.bin673 MB · 67%
At a glance
- Task
- Fill-Mask
- Library
- transformers
- Model type
- roberta
- Access
- Public
- Created
- Mar 2, 2022
- Updated
- May 20, 2021
- SHA
- 4ef4b1ef
Model card
From the Hugging Face model README
Welcome to KanBERTo (ಕನ್ಬರ್ಟೋ)
Model Description
This is a small language model for Kannada language with 1M data samples taken from OSCAR page
Training params
-
Dataset - 1M data samples are used to train this model from OSCAR page(https://traces1.inria.fr/oscar/) eventhough data set is of 1.7 GB due to resource constraint to train I have picked only 1M data from the total 1.7GB data set. If you are interested in collaboration and have computational resources to train on you are most welcome to do so.
-
Preprocessing - ByteLevelBPETokenizer is used to tokenize the sentences at character level and vocabulary size is set to 52k as per standard values given by 🤗
-
Hyperparameters - ByteLevelBPETokenizer : vocabulary size = 52_000 and min_frequency = 2 Trainer : num_train_epochs=12 - trained for 12 epochs per_gpu_train_batch_size=64 - batch size for the datasamples is 64 save_steps=10_000 - save model for every 10k steps save_total_limit=2 - save limit is set for 2
Intended uses & limitations this is for anyone who wants to make use of kannada language models for various tasks like language generation, translation and many more use cases.
Whatever else is helpful! If you are intersted in collaboration feel free to reach me Naveen
- Task
- Fill-Mask
- Library
- transformers
- Type
- roberta
- Languages
- kn
- Created
- Mar 2, 2022
- Updated
- May 20, 2021