Downloads · 30 days
0
ManipuriGPT-Tokenizer-v1
nanskong/ManipuriGPT-Tokenizer-v1
ManipuriGPT-Tokenizer-v1 is a machine learning model from nanskong. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for sentencepiece. The card lists the license as apache-2.0.
The official SentencePiece BPE tokenizer for the ManipuriGPT foundation model ecosystem, trained on the deduplicated ManipuriGPT-Corpus-v1.0.
Access
Public
Updated Jul 27, 2026
Repo size
649 KB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.model649 KB · 45%
At a glance
- Library
- sentencepiece
- License
- apache-2.0
- Access
- Public
- Created
- Jul 27, 2026
- Updated
- Jul 27, 2026
- SHA
- ec9ca39c
Model card
From the Hugging Face model README
ManipuriGPT-Tokenizer-v1.0
The official SentencePiece BPE tokenizer for the ManipuriGPT foundation model ecosystem, trained on the deduplicated ManipuriGPT-Corpus-v1.0.
Key Metrics
| Metric | Value |
|---|---|
| Algorithm | SentencePiece BPE |
| Vocabulary Size | 32,000 subwords |
| Vocabulary Utilization | 91.09% |
Unknown Token Rate (<unk>%) | 0.0000% |
| Avg Tokens / Sequence | 160.3 |
| Compression Ratio | 3.653 chars/token |
| Token Entropy | 11.64 bits |
Special Tokens
| Token | ID |
|---|---|
<pad> | 0 |
<unk> | 1 |
<s> | 2 |
</s> | 3 |
<meitei> | 4 |
<bengali> | 5 |
<romanized> | 6 |
<mask> | 7 |
Supported Scripts
- Meitei Mayek — Primary native Unicode script
- Bengali Script — Historical and legacy literary texts
- English / Latin — Romanized Manipuri and code-switching
Usage
import sentencepiece as spm
sp = spm.SentencePieceProcessor()
sp.Load("tokenizer.model")
text = "ꯃꯅꯤꯄꯨꯔꯤ ꯂꯣꯟ ꯑꯁꯤ ꯑꯆꯧꯕ ꯂꯣꯟ ꯑꯃꯅꯤ"
tokens = sp.Encode(text, out_type=str)
print(tokens)
# Decode back
decoded = sp.Decode(tokens)
print(decoded)
Files
| File | Description |
|---|---|
tokenizer.model | SentencePiece binary model (BPE) |
tokenizer.vocab | Full vocabulary with log-probabilities |
tokenizer_config.json | Training metadata, special tokens, and evaluation metrics |
special_tokens_map.json | HF-compatible special token mapping |
tokenizer_qualitative_samples.json | 50 qualitative encoding/decoding samples for manual inspection |
Training Details
- Training Corpus: ManipuriGPT-Corpus-v1.0 (147,065 deduplicated sequences)
- Character Coverage: 100%
- Byte Fallback: Enabled (zero
<unk>guarantee) - Script-Aware Special Tokens:
<meitei>,<bengali>,<romanized>for script-conditioned generation
Citation
@misc{manipurigpt_tokenizer_v10,
author = {ManipuriGPT Team},
title = {ManipuriGPT-Tokenizer-v1.0: SentencePiece BPE Tokenizer for Manipuri},
year = {2026},
publisher = {Hugging Face},
url = {https://huggingface.co/nanskong/ManipuriGPT-Tokenizer-v1}
}
Related Resources
- Library
- sentencepiece
- License
- apache-2.0
- Languages
- mni, en, bn
- Created
- Jul 27, 2026
- Updated
- Jul 27, 2026