Downloads · 30 days
16
1% of all-time downloads
lang-segmentation-roberta
nikitast/lang-segmentation-roberta
lang-segmentation-roberta is a token classification model from nikitast. Use it when you need labels on individual words, such as names. It is set up for transformers.
RoBERTa fine-tuned on small parts of Open Subtitles, Oscar and Tatoeba datasets (~9k samples per language).
All-time downloads
1.2K
Public
Repo size
2.2 GB
Likes
4
Public
Repo makeup
Click a slice to open those files.
.bin1.1 GB · 98%
At a glance
- Task
- Token Classification
- Library
- transformers
- Model type
- xlm-roberta
- Access
- Public
- Created
- May 12, 2022
- Updated
- Jan 7, 2023
- SHA
- b1aa48fa
Try a prompt
Datasets
Model card
From the Hugging Face model README
RoBERTa for Multilabel Language Segmentation
Training
RoBERTa fine-tuned on small parts of Open Subtitles, Oscar and Tatoeba datasets (~9k samples per language).
Implemented heuristic algorithm for multilingual training data creation with generation of target masks- https://github.com/n1kstep/lang-classifier
| data source | language |
|---|---|
| open_subtitles | ka, he, en, de |
| oscar | be, kk, az, hu |
| tatoeba | ru, uk |
Validation
The metrics obtained from validation on the another part of dataset (~1k samples per language).
| Validation Loss | Precision | Recall | F1-Score | Accuracy |
|---|---|---|---|---|
| 0.029172 | 0.919623 | 0.933586 | 0.926552 | 0.991883 |
- Task
- Token Classification
- Library
- transformers
- Type
- xlm-roberta
- Languages
- ru, uk, be, kk, az, hy
- Created
- May 12, 2022
- Updated
- Jan 7, 2023