Downloads · 30 days
9
0% of all-time downloads
mariav/bert-base-spanish-wwm-cased-finetuned-tweets
bert-base-spanish-wwm-cased-finetuned-tweets is a fill-mask model from mariav. Use it when you need the model to fill a missing word. It is set up for transformers. The card lists the license as apache-2.0.
should probably proofread and complete it, then remove this comment. --
Downloads · 30 days
9
0% of all-time downloads
All-time downloads
2.3K
Public
Repo size
5.7 GB
Likes
0
Public
Click a slice to open those files.
.bin440 MB · 100%
From the Hugging Face model README
This model is a fine-tuned version of bert-base-spanish-wwm-cased on the dataset petro-tweets. It achieves the following results on the evaluation set:
This model is a fine-tuned version of bert-base-spanish-wwm-cased using the dataset from petro-tweets (available in Huggin Face). The purpose is to extend the bert-base-spanish-cased domain, which, once fine-tuned, will be modified for the fill-in-the-gaps task.
The use is limited to school use and the limitations have to do with the size of the dataset, since it does not allow for a large contribution, a larger dataset would have to be used to get a larger contribution.
I did a training that gives the training and validation set loss. (It takes a lot of time. If you're using colab, I recommend to use less Epochs because the result does not change too much, and even though the loss is quite high, the performance of the model based on the perplexity is quite good) Also, I checked the perplexity, which is one good measure for Languages Models. As seen before, it gives a good perplexity, because it's quite low.
The following hyperparameters were used during training:
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 1.9867 | 1.0 | 79 | 2.8373 |
| 2.1251 | 2.0 | 158 | 2.8581 |
| 2.0998 | 3.0 | 237 | 2.7878 |
| 2.1207 | 4.0 | 316 | 2.8046 |
| 2.1268 | 5.0 | 395 | 2.7893 |
| 2.1453 | 6.0 | 474 | 2.8231 |
| 2.1548 | 7.0 | 553 | 2.7538 |
| 2.1582 | 8.0 | 632 | 2.7992 |
| 2.1775 | 9.0 | 711 | 2.6910 |
| 2.1962 | 10.0 | 790 | 2.7582 |
| 2.226 | 11.0 | 869 | 2.7252 |
| 2.2153 | 12.0 | 948 | 2.7437 |
| 2.2421 | 13.0 | 1027 | 2.7776 |
| 2.2297 | 14.0 | 1106 | 2.7257 |
| 2.2265 | 15.0 | 1185 | 2.7722 |
| 2.2187 | 16.0 | 1264 | 2.7475 |
| 2.2272 | 17.0 | 1343 | 2.7317 |
| 2.2315 | 18.0 | 1422 | 2.6876 |
| 2.2619 | 19.0 | 1501 | 2.7518 |
| 2.2313 | 20.0 | 1580 | 2.7373 |