Downloads · 30 days
0
PABITRA07/hindi-english-wordpiece-nmt
hindi-english-wordpiece-nmt is a machine learning model from PABITRA07. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers.
language: - en - hi license: apache-2.0 tags: - nmt - english - hindi - subword-tokenizer - wordpiece ---
Downloads · 30 days
0
Access
Public
Updated Sep 20, 2025
Repo size
—
Likes
0
Public
Click a slice to open those files.
.json794 KB · 100%
From the Hugging Face model README
language:
This repository contains a WordPiece tokenizer trained from scratch on a parallel English-Hindi corpus. It was developed as part of a research project to analyze the impact of different subword tokenization strategies on the performance and inference latency of a Neural Machine Translation (NMT) model.
This is a PreTrainedTokenizerFast object, wrapping a tokenizer from the tokenizers library.
~1.2 million sentences).[UNK], [PAD], [CLS], [SEP], [MASK]BertPreTokenizerThis tokenizer is intended to be used with a Transformer-based model (like the one from our research) for English-to-Hindi machine translation.
You can load and use this tokenizer directly in your Python code using the transformers library.
from transformers import PreTrainedTokenizerFast
# Load the tokenizer from the Hub
tokenizer = PreTrainedTokenizerFast.from_pretrained("sachin333/hindi-english-wordpiece-nmt")
# Example Usage
english_text = "This is a test of the tokenizer."
hindi_text = "यह टोकनाइज़र का एक परीक्षण है।"
encoded_en = tokenizer(english_text)
encoded_hi = tokenizer(hindi_text)
print("Encoded English:", encoded_en)
print("Decoded English:", tokenizer.decode(encoded_en['input_ids']))
print("\nEncoded Hindi:", encoded_hi)
print("Decoded Hindi:", tokenizer.decode(encoded_hi['input_ids']))