Downloads · 30 days
0
ag_hindi_bpe_tokenizer_32k
adityaghai07/ag_hindi_bpe_tokenizer_32k
ag_hindi_bpe_tokenizer_32k is a machine learning model from adityaghai07. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as mit.
Character-level BPE tokenizer for Hindi (Devanagari script) with Metaspace pre-tokenization. Trained on Hindi Wikipedia.
Access
Public
Updated Mar 15, 2026
Repo size
—
Likes
0
Public
Repo makeup
Click a slice to open those files.
.json2.8 MB · 100%
At a glance
- Library
- transformers
- License
- mit
- Access
- Public
- Created
- Mar 15, 2026
- Updated
- Mar 15, 2026
- SHA
- decd5289
Model card
From the Hugging Face model README
Hindi BPE Tokenizer (32K)
Character-level BPE tokenizer for Hindi (Devanagari script) with Metaspace pre-tokenization. Trained on Hindi Wikipedia.
- Vocab size: 32,000
- Algorithm: BPE with Metaspace
- Training data: Hindi Wikipedia (~519 MB, 688K lines)
- Normalization: Unicode NFC
- Special tokens:
<|pad|><|bos|><|eos|><|unk|><|sep|><|mask|><|user|><|assistant|>
Fertility ~1.5-1.8 tokens/word vs ~6-8 for GPT-2 on Hindi text. Preserves Devanagari conjuncts, matras, and combining marks. Handles code-mixed Hindi-English.
Usage
from transformers import PreTrainedTokenizerFast
tokenizer = PreTrainedTokenizerFast.from_pretrained("adityaghai07/ag_hindi_bpe_tokenizer_32k")
ids = tokenizer.encode("भारत एक महान देश है।")
print(tokenizer.decode(ids))
Source
- Library
- transformers
- License
- mit
- Languages
- hi
- Created
- Mar 15, 2026
- Updated
- Mar 15, 2026