Downloads · 30 days
0
bert-amharic-tokenizer
rasyosef/bert-amharic-tokenizer
bert-amharic-tokenizer is a machine learning model from rasyosef. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as mit.
This repo contains a WordPiece tokenizer trained on the Amharic subset of the oscar and mc4 datasets. It's the same as the BERT tokenizer but trained from scratch on an amharic dataset with a vocabulary size of 30522.
Access
Public
Updated Feb 8, 2024
Repo size
—
Likes
4
Public
Repo makeup
Click a slice to open those files.
.json825 KB · 70%
At a glance
- Library
- transformers
- License
- mit
- Access
- Public
- Created
- Feb 8, 2024
- Updated
- Feb 8, 2024
- SHA
- eb373ec0
Model card
From the Hugging Face model README
Amharic WordPiece Tokenizer
This repo contains a WordPiece tokenizer trained on the Amharic subset of the oscar and mc4 datasets. It's the same as the BERT tokenizer but trained from scratch on an amharic dataset with a vocabulary size of 30522.
How to use
You can load the tokenizer from huggingface hub as follows.
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("rasyosef/bert-amharic-tokenizer")
tokenizer.tokenize("የዓለምአቀፉ ነጻ ንግድ መስፋፋት ድህነትን ለማሸነፍ በሚደረገው ትግል አንዱ ጠቃሚ መሣሪያ ሊሆን መቻሉ ብዙ የሚነገርለት ጉዳይ ነው።")
Output:
['የዓለም', '##አቀፉ', 'ነጻ', 'ንግድ', 'መስፋፋት', 'ድህነትን', 'ለማሸነፍ', 'በሚደረገው', 'ትግል', 'አንዱ', 'ጠቃሚ', 'መሣሪያ', 'ሊሆን', 'መቻሉ', 'ብዙ', 'የሚነገርለት', 'ጉዳይ', 'ነው', '።']
- Library
- transformers
- License
- mit
- Languages
- am
- Created
- Feb 8, 2024
- Updated
- Feb 8, 2024