Hindi_Offline_Handwritten_OCR

Hindi OCR Model (Vit + Hindi RoBERTa)

Model Description

This model is a Vision Encoder-Decoder-based OCR model for recognizing Hindi text from images. The encoder uses a Vision Transformer (ViT) architecture, and the decoder is based on a pre-trained Hindi RoBERTa model. The model is capable of processing image inputs and generating corresponding text, specifically optimized for Hindi script.

Model Architecture

• Encoder: google/vit-base-patch16-224-in21k

  • A Vision Transformer (ViT) model pre-trained on ImageNet21k, used for encoding image inputs.

• Decoder: surajp/RoBERTa-hindi-guj-san

  • A RoBERTa model pre-trained on Hindi text data, fine-tuned to generate text based on visual features from the encoder.

Use Cases

The model is designed to perform Optical Character Recognition (OCR) on images containing Hindi text. Some potential use cases include:

• Extracting Hindi text word from scanned documents.

How to Use

You can use this model with Hugging Face's transformers library to extract text from images. Below is a sample usage script:

from PIL import Image
from transformers import AutoFeatureExtractor, AutoTokenizer, TrOCRProcessor, VisionEncoderDecoderModel

# Load the model and processor
encoder_model = 'google/vit-base-patch16-224-in21k'
decoder_model = 'surajp/RoBERTa-hindi-guj-san'
trained_model_path = '.model/'  # Path to the fine-tuned model

# Initialize the processor and model
feature_extractor = AutoFeatureExtractor.from_pretrained(encoder_model)
tokenizer = AutoTokenizer.from_pretrained(decoder_model)
processor = TrOCRProcessor(feature_extractor=feature_extractor, tokenizer=tokenizer)
model = VisionEncoderDecoderModel.from_pretrained(trained_model_path)

# Load and preprocess the image
image_path = 'path_to_your_image.jpg'
image = Image.open(image_path).convert('RGB')

# Generate text
pixel_values = processor(image, return_tensors='pt').pixel_values
generated_ids = model.generate(pixel_values)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]

print("Generated Text:", generated_text)

Inputs

• Image: The input is a single image containing Hindi text. Supported formats include .jpg, .png, and .jpeg.

Outputs

• Text: The model generates a string of text in Hindi extracted from the input image.

Example

# Input: Image containing Hindi text.
# Output: Extracted text from the image.

Training

• Dataset: The model was fine-tuned using a dataset of scanned Hindi hanndwritten text.
• Loss Function: Cross-Entropy Loss was used during training.
• Optimization: The Adam optimizer was employed with a learning rate of 5e-5.

Limitations

• Language Specificity: This model is optimized for Hindi script recognition. Performance on other languages or mixed-language documents may not be ideal.
• Image Quality: The model's performance is dependent on the quality of the input image. Images that are too blurry, noisy, or have poor lighting may produce less accurate results.
• Text Length: The model is optimized for extracting text with a maximum length of 64 characters. Longer texts might be truncated or inaccurately predicted.
• Small Text: The model may struggle with images containing very small or intricate fonts.

Evaluation

The model was evaluated using standard OCR benchmarks with an emphasis on Hindi text recognition. The primary evaluation metric was character-level accuracy and Word Error Rate (WER).

• Character Accuracy: Achieved ~73.2% accuracy on validation sets.
• Train Loss: 0.063800
• Validation Loss: 0.201061
• CER: 0.101732
• Test CER :0.118332

Ethical Considerations

This model, while useful for Hindi text extraction, should be applied with caution in contexts where incorrect text extraction could lead to harmful outcomes, such as legal or medical document analysis.

License

This model is distributed under the MIT license. Please check the Hugging Face repository for specific terms.

Live Demo

A live demo of the Hindi Offline Handwritten OCR can be found here:
Hindi Offline Handwritten OCR on Hugging Face Spaces