Downloads · 30 days
10
15% of all-time downloads
indian-slm-hindi-30m
way2hemanthkumar/indian-slm-hindi-30m
indian-slm-hindi-30m is a machine learning model from way2hemanthkumar. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
A LLaMA-style decoder-only language model trained from scratch on Hindi text.
All-time downloads
67
Public
Parameters
46.9M
189 MB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors188 MB · 99%
Parameter types
How the weights are stored.
F3246.9M · 100%
Model card
From the Hugging Face model README
Indian SLM — Hindi Foundational Model
A LLaMA-style decoder-only language model trained from scratch on Hindi text.
Model Details
| Property | Value |
|---|---|
| Architecture | LLaMA-style (RMSNorm, RoPE, GQA, SwiGLU) |
| Parameters | ~31M |
| Layers | 6 |
| Hidden dim | 512 |
| Attention heads | 8 Q / 4 KV (GQA) |
| FFN hidden dim | 1024 |
| Vocab size | 32,000 |
| Max seq length | 512 |
| Training steps | 400 |
| Dataset | Hindi Wikipedia (wikimedia/wikipedia 20231101.hi) |
| Tokenizer | SentencePiece BPE (32k vocab) |
Architecture
- RMSNorm instead of LayerNorm
- RoPE (Rotary Position Embeddings) for position encoding
- GQA (Grouped Query Attention) — 8 Q heads, 4 KV heads
- SwiGLU feed-forward activation
- Weight tying between embedding and LM head
Evaluation (base model, no fine-tuning)
| Metric | Value |
|---|---|
| Overall Perplexity | 391 |
| Top-1 Accuracy | 20.75% |
| Top-5 Accuracy | 32.37% |
| Vocab Coverage | 100% |
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("way2hemanthkumar/indian-slm-hindi-30m")
model = AutoModelForCausalLM.from_pretrained("way2hemanthkumar/indian-slm-hindi-30m")
inputs = tokenizer("भारत एक", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(output[0]))
Limitations
This is a base model trained for validation purposes on a small dataset (500 steps). It is not instruction-tuned. Outputs may be incoherent. Fine-tuning is required for practical use.
Training Details
- Optimizer: AdamW (β1=0.9, β2=0.95)
- LR schedule: linear warmup + cosine decay
- Gradient clipping: 1.0
- Gradient accumulation: 4 steps
- Type
- indian_slm
- License
- apache-2.0
- Languages
- hi, en
- Created
- Mar 28, 2026
- Updated
- Mar 28, 2026