Downloads · 30 days
0
bert-amharic-tokenizer-24k
rasyosef/bert-amharic-tokenizer-24k
bert-amharic-tokenizer-24k is a machine learning model from rasyosef. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as mit.
This repo contains a WordPiece tokenizer trained on the Amharic subset of the oscar and mc4 datasets. It's the same as the BERT tokenizer but trained from scratch on an amharic text dataset, with a vocabulary size of…
Access
Public
Updated May 10, 2024
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json609 KB · 73%
At a glance
- Library
- transformers
- License
- mit
- Access
- Public
- Created
- Apr 17, 2024
- Updated
- May 10, 2024
- SHA
- b5448dbf
Model card
From the Hugging Face model README
Amharic WordPiece Tokenizer
This repo contains a WordPiece tokenizer trained on the Amharic subset of the oscar and mc4 datasets. It's the same as the BERT tokenizer but trained from scratch on an amharic text dataset, with a vocabulary size of 24576.
How to use
You can load the tokenizer from huggingface hub as follows.
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("rasyosef/bert-amharic-tokenizer")
tokenizer.tokenize("የዓለምአቀፉ ነጻ ንግድ መስፋፋት ድህነትን ለማሸነፍ በሚደረገው ትግል አንዱ ጠቃሚ መሣሪያ ሊሆን መቻሉ ብዙ የሚነገርለት ጉዳይ ነው።")
Output:
['የዓለም', '##አ', '##ቀፉ', 'ነጻ', 'ንግድ', 'መስፋፋት', 'ድህነትን', 'ለማሸነፍ', 'በሚደረገው', 'ትግል', 'አንዱ', 'ጠቃሚ', 'መሣሪያ', 'ሊሆን', 'መቻሉ', 'ብዙ', 'የሚነገር', '##ለት', 'ጉዳይ', 'ነው', '።']
- Library
- transformers
- License
- mit
- Languages
- am
- Created
- Apr 17, 2024
- Updated
- May 10, 2024