Prakram-Minibert

Prakram-MiniBERT

Prakram-MiniBERT — A small BERT-style masked language model (MiniBERT) trained from scratch on Hindi & Sanskrit corpora.

---

Model description

Model type: BERT-style encoder (Masked Language Model).
Model size / shape: MiniBERT — num_hidden_layers=4, hidden_size=256, num_attention_heads=4, intermediate_size=1024.
Tokenizer: WordPiece tokenizer trained on Devanagari corpus (Hindi + Sanskrit).
Vocabulary size: <fill_here from tokenizer.vocab_size> (please see tokenizer/vocab.txt).
Tasks: Text understanding tasks such as classification, named-entity recognition, question answering (when fine-tuned), and masked token prediction / fill-mask. Also useful for embedding extraction after fine-tuning.

---

Intended use & applications

This model is meant to be a compact, open foundation model for downstream NLP tasks in Hindi and Sanskrit:

• Building chatbots / conversational agents in Hindi / Sanskrit (by fine-tuning).
• Question-answering systems using domain corpora (fine-tune on SQuAD-style data).
• Text classification (sentiment, topic, etc).
• Masked token prediction and linguistic analysis of Sanskrit/Hindi text.
• Educational tools, cultural heritage processing.

Not recommended for: high-sensitivity tasks without further fine-tuning and validation (medical, legal, safety-critical decisions). See Limitations below.

---

How the model was trained

• Objective: Masked Language Modeling (MLM).
• Tokenizer: WordPiece trained on a combined Devanagari corpus (Hindi + Sanskrit).
• Corpus: newcorpora.txt (user-provided corpus of Hindi and Sanskrit texts).
• Training compute: Google Colab (T4 GPU).
• Training recipe: Hugging Face Trainer with DataCollatorForLanguageModeling and standard AdamW optimizer.
• Config: model configuration saved in model_config/config.json.

Note: This repo contains the model weights, tokenizer files, and configuration. If you used any private or copyrighted data during training, be careful about licensing and distribution.

---

Evaluation

• Evaluation metric reported: Masked LM loss and approximate perplexity.
• Example (fill values after you run evaluation):
  • eval_loss: <fill_eval_loss>
  • perplexity: <fill_perplexity>

Please run the included evaluation script in this repo (or in Colab) to compute final metrics on your validation split.

---

How to use (examples)

Installation

pip install transformers