Downloads · 30 days
124
14% of all-time downloads
bert-base-sw
orai-nlp/bert-base-sw
bert-base-sw is a feature extraction model from orai-nlp. Use it when you need embeddings to search or compare text. It is set up for transformers. The card lists the license as cc-by-4.0.
BERT base (cased) model trained on a subset of 125M tokens of cc100-Swahili for our work Scaling Laws for BERT in Low-Resource Settings at ACL2023 Findings.
All-time downloads
859
Public
Parameters
125M
2.5 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.bin501 MB · 50%
Parameter types
How the weights are stored.
F32125M · 100%
Model card
From the Hugging Face model README
BERT base (cased) model trained on a subset of 125M tokens of cc100-Swahili for our work Scaling Laws for BERT in Low-Resource Settings at ACL2023 Findings.
The model has 124M parameters (12L), and a vocab size of 50K. It was trained for 500K steps with a sequence length of 512 tokens and batch-size of 256.
Results
| bert-base-sw | bert-medium-sw | Flair | mBERT | SwahBERT | |
|---|---|---|---|---|---|
| NERC | 92.09 | 91.63 | 92.04 | 91.17 | 88.60 |
| Topic | 93.07 | 92.88 | 91.83 | 91.52 | 90.90 |
| Sentiment | 79.04 | 77.07 | 73.60 | 69.17 | 71.12 |
| QNLI | 63.34 | 63.87 | 52.82 | 63.48 | 64.72 |
Authors
Gorka Urbizu [1], Iñaki San Vicente [1], Xabier Saralegi [1], Rodrigo Agerri [2] and Aitor Soroa [2]
Affiliation of the authors:
[1] Orai NLP Technologies
[2] HiTZ Center - Ixa, University of the Basque Country UPV/EHU
Licensing
The model is licensed under the Creative Commons Attribution 4.0. International License (CC BY 4.0).
To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
Acknowledgements
If you use this model please cite the following paper:
- G. Urbizu, I. San Vicente, X. Saralegi, R. Agerri, A. Soroa. Scaling Laws for BERT in Low-Resource Settings. Findings of the Association for Computational Linguistics: ACL 2023. July, 2023. Toronto, Canada
Contact information
Gorka Urbizu, Iñaki San Vicente: {g.urbizu,i.sanvicente}@orai.eus
- Task
- Feature Extraction
- Library
- transformers
- Type
- bert
- License
- cc-by-4.0
- Languages
- sw
- Created
- May 12, 2023
- Updated
- Feb 7, 2025