dhivehi-roberta-tokenizer-extended

Dhivehi-Roberta Tokenizer (Extended Vocabulary)

This repository contains a custom extension of the roberta-base tokenizer, enhanced with 10,000 frequently used Dhivehi tokens.

Overview

• Base tokenizer: roberta-base (vocab size: 50265)
• Dhivehi tokens added: 10,000
• Source: Top 424,319 unique Dhivehi tokens (filtered to 3+ characters)
• Use case: Improves tokenization of Dhivehi text for models like TrOCR, mBERT, or RoBERTa.

Comparison Results

On Dhivehi Text: އީދުގެ ހަރަކާތް

🔹 Stock roberta-base

Tokens: ['Þ', 'ĩ', 'Þ', '©', 'Þ', 'ĭ', 'Þ', 'ª', 'Þ', 'İ', 'Þ', '¬', 'Ġ', ...]
Token IDs: [49944, 6382, 49944, 15375, 49944, 13859, ...]

🔸 Custom Dhivehi Extended

Tokens: ['އީދުގެ', 'Ġ', 'ހަރަކާތް']
Token IDs: [59825, 1437, 53039]

The extended tokenizer produces clean, language-aware tokens from valid Thaana characters, unlike the corrupted outputs from the base tokenizer.

On English Text (No Change Expected)

Tokens: ['The', 'Ġquick', 'Ġbrown', 'Ġfox', 'Ġjumps', 'Ġover', 'Ġthe', 'Ġlazy', 'Ġdog', '.']
Token IDs: [133, 2119, 6219, 23602, 13855, 81, 5, 22414, 2335, 4]

English performance remains identical to the original roberta-base.

How to Use

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("alakxender/dhivehi-roberta-tokenizer-extended")
tokens = tokenizer.tokenize("އީދުގެ ހަރަކާތް")
print(tokens)