Downloads · 30 days
0
ag_hindi_uni_tokenizer_32k
adityaghai07/ag_hindi_uni_tokenizer_32k
ag_hindi_uni_tokenizer_32k is a machine learning model from adityaghai07. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for sentencepiece. The card lists the license as mit.
Unigram language model tokenizer for Hindi (Devanagari script) using SentencePiece. Trained on Hindi Wikipedia.
Access
Public
Updated Mar 15, 2026
Repo size
1.1 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.model1.1 MB · 54%
At a glance
- Library
- sentencepiece
- License
- mit
- Access
- Public
- Created
- Mar 15, 2026
- Updated
- Mar 15, 2026
- SHA
- 1b798d35
Model card
From the Hugging Face model README
Hindi Unigram Tokenizer (32K)
Unigram language model tokenizer for Hindi (Devanagari script) using SentencePiece. Trained on Hindi Wikipedia.
- Vocab size: 32,000
- Algorithm: Unigram (SentencePiece)
- Training data: Hindi Wikipedia (~519 MB, 688K lines)
- Character coverage: 0.9995
- Normalization: nmt_nfkc
Fertility ~1.8-2.0 tokens/word. Probabilistic tokenization supports subword regularization during training.
Usage
import sentencepiece as spm
sp = spm.SentencePieceProcessor()
sp.Load("hindi_unigram.model")
tokens = sp.EncodeAsPieces("भारत एक महान देश है।")
ids = sp.EncodeAsIds("भारत एक महान देश है।")
print(sp.DecodeIds(ids))
From HuggingFace Hub
from huggingface_hub import hf_hub_download
import sentencepiece as spm
model_path = hf_hub_download(
repo_id="adityaghai07/ag_hindi_uni_tokenizer_32k",
filename="hindi_unigram.model"
)
sp = spm.SentencePieceProcessor()
sp.Load(model_path)
Source
- Library
- sentencepiece
- License
- mit
- Languages
- hi
- Created
- Mar 15, 2026
- Updated
- Mar 15, 2026