Downloads · 30 days
6
12% of all-time downloads
monolingual-hindi-from-scratch
ishathombre/monolingual-hindi-from-scratch
monolingual-hindi-from-scratch is a fill-mask model from ishathombre. Use it when you need the model to fill a missing word. It is set up for transformers.
BERT model trained from scratch using a custom tokenizer with a 64,000-token vocabulary.
All-time downloads
49
Public
Parameters
24.9M
298 MB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.pt199 MB · 66%
At a glance
- Task
- Fill-Mask
- Library
- transformers
- Model type
- bert
- Access
- Public
- Created
- Jun 2, 2025
- Updated
- Jun 6, 2025
- SHA
- 1616c2b6
Base models
Datasets
Model card
From the Hugging Face model README
BERT from Scratch (1 Epoch, Training Loss: 4.13)
BERT model trained from scratch using a custom tokenizer with a 64,000-token vocabulary.
- Training: 1 epoch
- Masked Language Modeling (MLM) loss: 4.13
- Tokenizer: Custom-trained, vocab size, on iit-madras Hindi-monolingual dataset = 64,000
- Architecture: Maximum position embeddings: 512 Hidden size: 312 Number of attention heads: 12 Number of transformer layers: 4 Intermediate (feed-forward) size: 1200 Type vocabulary size: 2 (for segment embeddings)
It is uploaded for checkpointing, experimentation, and community feedback.
Intended Use
- Research on training dynamics
- Continued pretraining
- Fine-tuning for downstream tasks (with caution)
Limitations
- Low training coverage (1 epoch)
- Not yet evaluated on downstream tasks
- Task
- Fill-Mask
- Library
- transformers
- Type
- bert
- Languages
- hi
- Created
- Jun 2, 2025
- Updated
- Jun 6, 2025