Downloads · 30 days
0
bangla-tokenizer
yasserius/bangla-tokenizer
bangla-tokenizer is a machine learning model from yasserius. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
A specialized Bangla/Bengali tokenizer extracted from multilingual models and extended with missing characters.
Access
Public
Updated Aug 31, 2025
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json1.4 MB · 78%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- Aug 31, 2025
- Updated
- Aug 31, 2025
- SHA
- 3dd15013
Model card
From the Hugging Face model README
Custom Bangla Tokenizer
A specialized Bangla/Bengali tokenizer extracted from multilingual models and extended with missing characters.
Features
- Focused on Bangla text tokenization
- Support for characters like ঢ় and other missing characters
- Reduced vocabulary size: 21,607 tokens
- Compatible with Hugging Face Transformers
Usage
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("yasserius/bangla-tokenizer")
# Tokenize Bangla text
text = "আমি বাংলায় কথা বলি"
tokens = tokenizer.tokenize(text)
print(tokens)
Model Details
- Base model: Extracted from google/muril-base-cased
- Language: Bengali/Bangla
- Vocabulary size: 21,607
- License
- apache-2.0
- Languages
- bn
- Created
- Aug 31, 2025
- Updated
- Aug 31, 2025