Downloads · 30 days
0
amharic_tokenizer
BiniyamAjaw/amharic_tokenizer
amharic_tokenizer is a machine learning model from BiniyamAjaw. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
- Vocabulary Size: 100,000 - Tokenizer Type: Byte-Pair Encoder
Access
Public
Updated Mar 20, 2024
Repo size
—
Likes
2
Public
Repo makeup
Click a slice to open those files.
.json5.2 MB · 100%
At a glance
- License
- mit
- Access
- Public
- Created
- Feb 1, 2024
- Updated
- Mar 20, 2024
- SHA
- 427c9dd8
Datasets
Model card
From the Hugging Face model README
Amharic Tokenizer
<!-- The model is trained on a vast amharic data to tokenize unseen data into tokens. -->Model Details
- Vocabulary Size: 100,000
- Tokenizer Type: Byte-Pair Encoder
Model Description
<!-- Tokenizer that uses BPE -->- Developed by: Biniyam Ajaw
- Language(s) (NLP): Amharic and Amharic-Driven Languages
- License: MIT
Model Sources [optional]
Uses
Model can be called by the autotokenizer module from the transformers package and can be used to tokenize any amharic text perfectly
- License
- mit
- Languages
- am
- Created
- Feb 1, 2024
- Updated
- Mar 20, 2024