Downloads · 30 days
338
0% of all-time downloads
BERTino
indigo-ai/BERTino
BERTino is a fill-mask model from indigo-ai. Use it when you need the model to fill a missing word. It is set up for transformers. The card lists the license as mit.
This repository hosts BERTino, an Italian DistilBERT model pre-trained by indigo.ai on a large general-domain Italian corpus. BERTino is task-agnostic and can be fine-tuned for every downstream task. The pre-training…
All-time downloads
224K
Public
Repo size
819 MB
Likes
17
Public
Repo makeup
Click a slice to open those files.
.bin273 MB · 50%
At a glance
- Task
- Fill-Mask
- Library
- transformers
- License
- mit
- Model type
- distilbert
- Access
- Public
- Created
- Mar 2, 2022
- Updated
- Sep 22, 2021
- SHA
- 4539d128
Try a prompt
Model card
From the Hugging Face model README
BERTino: an Italian DistilBERT model
This repository hosts BERTino, an Italian DistilBERT model pre-trained by indigo.ai on a large general-domain Italian corpus. BERTino is task-agnostic and can be fine-tuned for every downstream task.
Corpus
The pre-training corpus that we used is the union of the Paisa and ItWaC corpora. The final corpus counts 14 millions of sentences for a total of 12 GB of text.
Downstream Results
To validate the pre-training that we conducted, we evaluated BERTino on the Italian ParTUT, Italian ISDT, Italian WikiNER and multi-class sentence classification tasks. We report for comparison results obtained by the teacher model fine-tuned in the same tasks and for the same number of epochs.
Italian ISDT:
| Model | F1 score | Fine-tuning time | Evaluation time |
|---|---|---|---|
| BERTino | 0,9801 | 9m, 4s | 3s |
| Teacher | 0,983 | 16m, 28s | 5s |
Italian ParTUT:
| Model | F1 score | Fine-tuning time | Evaluation time |
|---|---|---|---|
| BERTino | 0,9268 | 1m, 18s | 1s |
| Teacher | 0,9688 | 2m, 18s | 1s |
Italian WikiNER:
| Model | F1 score | Fine-tuning time | Evaluation time |
|---|---|---|---|
| BERTino | 0,9038 | 35m, 35s | 3m, 1s |
| Teacher | 0,9178 | 67m, 8s | 5m, 16s |
Multi-class sentence classification:
| Model | F1 score | Fine-tuning time | Evaluation time |
|---|---|---|---|
| BERTino | 0,7788 | 4m, 40s | 6s |
| Teacher | 0,7986 | 8m, 52s | 9s |
- Task
- Fill-Mask
- Library
- transformers
- Type
- distilbert
- License
- mit
- Languages
- it
- Created
- Mar 2, 2022
- Updated
- Sep 22, 2021