Downloads · 30 days
0
malayalam-bpe-tokenizer
smcproject/malayalam-bpe-tokenizer
malayalam-bpe-tokenizer is a machine learning model from smcproject. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for tokenizers. The card lists the license as mit.
A Byte Pair Encoding (BPE) tokenizer trained on Malayalam text corpus. Trained using the HuggingFace tokenizers library with Metaspace pre-tokenization and NFC normalization for correct handling of Malayalam Unicode c…
Access
Public
Updated Feb 21, 2026
Repo size
—
Likes
3
Public
Repo makeup
Click a slice to open those files.
.json1.1 MB · 100%
At a glance
- Library
- tokenizers
- License
- mit
- Access
- Public
- Created
- Feb 21, 2026
- Updated
- Feb 21, 2026
- SHA
- 8f93d1a7
Model card
From the Hugging Face model README
Malayalam BPE Tokenizer
A Byte Pair Encoding (BPE) tokenizer trained on Malayalam text corpus. Trained using the HuggingFace tokenizers library with Metaspace pre-tokenization and NFC normalization for correct handling of Malayalam Unicode conjuncts.
Details
| Property | Value |
|---|---|
| Algorithm | BPE (Byte Pair Encoding) |
| Vocabulary size | 16,000 |
| Pre-tokenizer | Metaspace (▁) |
| Normalizer | NFC + Strip |
| Special tokens | <s>, </s>, <unk>, <pad>, <mask> |
Usage
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("smc/malayalam-bpe-tokenizer")
text = "മലയാളം ഒരു ദ്രാവിഡ ഭാഷയാണ്"
tokens = tokenizer.tokenize(text)
print(tokens)
encoded = tokenizer(text, return_tensors="pt")
print(encoded)
Notes
- Use Metaspace (not ByteLevel) pre-tokenization — ByteLevel splits Malayalam's multibyte UTF-8 sequences into invalid bytes.
- NFC normalization ensures Malayalam conjuncts formed via ZWJ/ZWNJ are handled consistently.
- Trained and published from smc/malayalam-tokenizer.
- Library
- tokenizers
- License
- mit
- Languages
- ml
- Created
- Feb 21, 2026
- Updated
- Feb 21, 2026