maithili-clm-124m

kamal-018/maithili-clm-124m

maithili-clm-124m is a machine learning model from kamal-018. Use it for the machine learning task on the model card, and read the license before you ship it in a product.

A decoder-only transformer pretrained from scratch on the Maithili language corpus.

Downloads · 30 days

3% of all-time downloads

All-time downloads

121

Public

Repo size

34.1 GB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.pt1.9 GB · 100%

At a glance

Access: Public
Created: Jul 26, 2026
Updated: Aug 8, 2026
SHA: 6763db19

Datasets

AiventraLab/maithili-corpus

Model card

From the Hugging Face model README

Maithili Causal Language Model (109.5M)

A decoder-only transformer pretrained from scratch on the Maithili language corpus.

Architecture

Component	Detail
Type	Decoder-only causal transformer
Parameters	~109.5M
Layers	12
Hidden dim	768
Attention heads	12
Head dim	64
Position encoding	RoPE (Rotary Position Embedding)
Normalization	RMSNorm
Feed-forward	SwiGLU
Vocab size	32,000
Max sequence length	1024
Tokenizer	Byte-level BPE (purpose-built for Maithili)

Training

Detail	Value
Training data	`AiventraLab/maithili-corpus` (Maithili gold corpus)
Total training tokens	~983,040,000
Steps	30000
Effective batch size	32,768 tokens
Precision	fp16 mixed precision
Optimizer	AdamW (betas=0.9/0.95, weight_decay=0.1)
LR schedule	Cosine with linear warmup (500 steps)
Peak LR	3e-4
Gradient checkpointing	Yes
Hardware	1x NVIDIA T4 (16GB)

Evaluation

Metric	Value
Final validation loss	3.1215
Final validation perplexity	22.68

Validation Loss Over Training

[EVAL] step 26000: val_loss=3.1731, val_ppl=23.88 [EVAL] step 27000: val_loss=3.2049, val_ppl=24.65 [EVAL] step 28000: val_loss=3.2059, val_ppl=24.68 [EVAL] step 29000: val_loss=3.2060, val_ppl=24.68 [EVAL] step 30000: val_loss=3.1897, val_ppl=24.28

Languages: mai
Created: Jul 26, 2026
Updated: Aug 8, 2026