hindi-tokenizer

Hindi-English BPE Tokenizer

Load and use the pre-trained Byte-Pair Encoding (BPE) tokenizer directly from Hugging Face Hub under DumbsterDrekk/hindi-tokenizer.

Installation

pip install transformers

(If your repo is private, also install and login to huggingface_hub)

pip install huggingface_hub
huggingface-cli login

Usage

1. Set your HF token (if private repo):

   export HF_TOKEN="hf_yourPersonalAccessToken"
   

2. Load the tokenizer:

   import os
   from transformers import PreTrainedTokenizerFast
   
   hf_token = os.getenv("HF_TOKEN")  # or None for public repo
   
   tokenizer = PreTrainedTokenizerFast.from_pretrained(
       "DumbsterDrekk/hindi-tokenizer",
       use_auth_token=hf_token,    # omit if repo is public
       unk_token="<unk>",
       eos_token="</w>",
       pad_token="<pad>",
   )
   

3. Tokenize text:

   text = "मैं आज बहुत खुश हूँ। I’m very happy today."
   
   # Single sequence
   enc = tokenizer(
       text,
       padding="max_length",
       truncation=True,
       max_length=512,
       return_tensors="pt",      # or omit for Python lists
   )
   
   print("Input IDs:     ", enc["input_ids"])
   print("Attention Mask:", enc["attention_mask"])
   

4. Decode tokens back to text:

   decoded = tokenizer.decode(enc["input_ids"][0], skip_special_tokens=True)
   print("Decoded Text:  ", decoded)
   

---

Once loaded, this tokenizer can be plugged into any 🤗 Transformers training or generation pipeline exactly like a built-in model tokenizer.