telugu-bpe-23k

🔡 Telugu BPE Tokenizer (23k vocab) — Vipplav

A Byte-Pair Encoding (BPE) tokenizer trained on over **3.4 lakh cleaned Telugu text keys ** from the AI4Bharat Sangraha dataset and other open sources. This tokenizer is ideal for pretraining or fine-tuning Telugu language models.

---

📌 Highlights

• Tokenizer Type: SentencePiece BPE
• Vocabulary Size: 23,000
• Character Coverage: 100% Telugu script
• Library: 🤗 transformers + sentencepiece
• Special Tokens:
  • <unk> — Unknown token
  • <pad> — Padding
  • <s> — Start of sequence
  • </s> — End of sequence
  • \n, ₹, •, - — User-defined symbols preserved during training

---

✨ Example Usage

from transformers import T5Tokenizer

# Load tokenizer from Hugging Face Hub
tokenizer = T5Tokenizer.from_pretrained("Vipplav/telugu-bpe-23k")

# Sample Telugu input
text = "పరిశీలన తేదీ: 15-06-2025"

# Tokenize the input
tokens = tokenizer.tokenize(text)

# Decode tokens back to text
decoded = tokenizer.decode(tokenizer.convert_tokens_to_ids(tokens), skip_special_tokens=True)

# Display results
print(f"\n📥 Input   : {text}")
print(f"🔤 Tokens  : {tokens}")
print(f"📝 Decoded : {decoded}")

📜 Citation

If you use this tokenizer, please cite:

APA:

Vipplav AI (2025). Telugu BPE Tokenizer (23k vocab). Hugging Face. https://huggingface.co/Vipplav/telugu-bpe-23k
AI4Bharat. (2023). Sangraha: A Large-Scale Multidomain Corpus for Indian Languages. Hugging Face Datasets. https://huggingface.co/datasets/ai4bharat/sangraha

BibTeX:

@misc{vipplav_telugu_tokenizer,
  author = {Vipplav AI},
  title = {Telugu BPE Tokenizer (23k vocab)},
  year = {2025},
  url = {https://huggingface.co/Vipplav/telugu-bpe-23k}
}
@dataset{sangraha2023,
  author = {AI4Bharat},
  title = {Sangraha Dataset},
  year = {2023},
  url = {https://huggingface.co/datasets/ai4bharat/sangraha}
}