Hindi_Tokenizer

Hindi Tokenizer Model Summary

Overview

This model card summarizes the Hindi tokenizer models produced from the training pipeline. The released artifacts include multiple tokenizer variants trained and evaluated on a Hindi corpus.

Supported Language

• Language: Hindi
• ISO code: hi
• Script: Devanagari

Training Data

The tokenizers were trained on Hindi text derived from source documents and processed through the pipeline for corpus cleaning and subword discovery.

Tokenizer Variants Produced

The following tokenizer artifacts were generated:

• custom_bpe_model.json
• sentencepiece_bpe.model
• sentencepiece_bpe.vocab
• sentencepiece_unigram.model
• sentencepiece_unigram.vocab
• comparison_report.json

Evaluation Summary

The evaluation report shows the following metrics for the trained tokenizers on 158,106 evaluated words:

Testing the Tokenizer

You can test the trained tokenizer models with sample Hindi text using the following Python snippet:

from pathlib import Path
import sys

PROJECT_ROOT = Path(__file__).resolve().parent
sys.path.insert(0, str(PROJECT_ROOT))

from pipelines.phase3.bpe_trainer import BPEModel
from pipelines.phase3.sentencepiece_trainer import load_sentencepiece


def load_tokenizers():
    phase3_dir = PROJECT_ROOT / "outputs" / "phase3"

    custom_bpe = BPEModel.load(phase3_dir / "custom_bpe_model.json")
    sp_bpe = load_sentencepiece(str(phase3_dir / "sentencepiece_bpe"))
    sp_unigram = load_sentencepiece(str(phase3_dir / "sentencepiece_unigram"))

    return custom_bpe, sp_bpe, sp_unigram


def tokenize_with_custom_bpe(model, samples):
    results = []
    for sample in samples:
        words = sample.split()
        tokens = []
        for word in words:
            tokens.extend(model.encode_word(word))
        results.append(tokens)
    return results


def tokenize_with_sentencepiece(sp, samples):
    return [sp.encode(sample, out_type=str) for sample in samples]


sample_texts = [
    "भारत",
    "भारतीय संविधान",
    "सरकार ने निर्णय लिया",
    "मजदूरी और शिक्षा",
    "विद्यालय में पढ़ाई होती है",
]

custom_bpe, sp_bpe, sp_unigram = load_tokenizers()

print("Custom BPE:")
for sample, tokens in zip(sample_texts, tokenize_with_custom_bpe(custom_bpe, sample_texts)):
    print(f"{sample} -> {tokens}")

print("\nSentencePiece BPE:")
for sample, tokens in zip(sample_texts, tokenize_with_sentencepiece(sp_bpe, sample_texts)):
    print(f"{sample} -> {tokens}")

print("\nSentencePiece Unigram:")
for sample, tokens in zip(sample_texts, tokenize_with_sentencepiece(sp_unigram, sample_texts)):
    print(f"{sample} -> {tokens}")

Notes

• The tokenizer pipeline uses classical/statistical methods rather than LLM APIs.
• The target vocabulary size used during training was 2000.