Downloads · 30 days
0
xlmr-dhivehi-tokenizer-extended
alakxender/xlmr-dhivehi-tokenizer-extended
xlmr-dhivehi-tokenizer-extended is a machine learning model from alakxender. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as mit.
This tokenizer extends xlm-roberta-base with 52,000 additional tokens derived from a large Dhivehi corpus. It significantly improves tokenization and decoding of Thaana script (Dhivehi language) content.
Access
Public
Updated Jun 12, 2025
Repo size
23.3 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json30.3 MB · 100%
At a glance
- Library
- transformers
- License
- mit
- Access
- Public
- Created
- Jun 9, 2025
- Updated
- Jun 12, 2025
- SHA
- 1cf58ac9
Model card
From the Hugging Face model README
Dhivehi-Extended XLM-RoBERTa Tokenizer
This tokenizer extends xlm-roberta-base with 52,000 additional tokens derived from a large Dhivehi corpus. It significantly improves tokenization and decoding of Thaana script (Dhivehi language) content.
What’s Included
- Base model:
xlm-roberta-base - 52,000 high-frequency Dhivehi tokens added using SentencePiece
- Custom
added_tokens.jsonfile - Improved Thaana tokenization and decoding
Tokenization Example
Stock XLM-R
Text: އީދުގެ ހަރަކާތް ...
Tokens: ['▁', '<unk>', ...]
Decoded: <unk> <unk> ...
Extended Tokenizer
Tokens: ['އީދު', 'ގެ', ' ހަރަކާތް', ...]
Decoded: އީދުގެ ހަރަކާތް ...
Notes
This tokenizer is best suited for Dhivehi tasks like:
- Named Entity Recognition (NER)
- Translation
- Language Modeling
English and other XLM-R languages remain fully compatible.
- Library
- transformers
- License
- mit
- Languages
- dv
- Created
- Jun 9, 2025
- Updated
- Jun 12, 2025