Downloads · 30 days
0
Benjamin-png/bert-tokenizer-swahili_25000_minfreq_1
bert-tokenizer-swahili_25000_minfreq_1 is a machine learning model from Benjamin-png. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
A BERT WordPiece tokenizer specifically trained for the Swahili language. This tokenizer is designed to provide effective tokenization for Swahili text, supporting BERT-based models and other transformer architectures.
Downloads · 30 days
0
Access
Public
Updated Feb 12, 2025
Repo size
—
Likes
0
Public
Click a slice to open those files.
.json571 KB · 99%
From the Hugging Face model README
A BERT WordPiece tokenizer specifically trained for the Swahili language. This tokenizer is designed to provide effective tokenization for Swahili text, supporting BERT-based models and other transformer architectures.
from transformers import AutoTokenizer
# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained("Benjamin-png/bert-tokenizer-swahili_25000_minfreq_1")
# Example usage
text = "Habari za asubuhi"
encoded = tokenizer(text)
print(encoded.tokens)
The tokenizer was trained with the following specifications:
Input: "Habari za asubuhi"
Tokens: ['[CLS]', 'habari', 'za', 'asubuhi', '[SEP]']
Input: "Ninafurahi kukutana nawe"
Tokens: ['[CLS]', 'ninafurahi', 'kukutana', 'nawe', '[SEP]']
Input: "Karibu Tanzania"
Tokens: ['[CLS]', 'karibu', 'tanzania', '[SEP]']
This tokenizer is designed for:
If you use this tokenizer in your research, please cite:
@misc{swahili-bert-tokenizer,
author = {Benjamin-png},
title = {BERT WordPiece Tokenizer for Swahili},
year = {2025},
publisher = {HuggingFace},
howpublished = {\url{https://huggingface.co/Benjamin-png/bert-tokenizer-swahili}}
}
For questions and feedback, please open an issue in the GitHub repository or contact through Hugging Face.
MIT License