gpt2-ne-wikimulti

GPT-2 Nepali WikiMulti

GPT-2-small model pretrained from scratch on the Nepali portion of WikiMulti.

Dataset

• Dataset: dehanalkautsar/WikiMulti
• File: 20260801_ne_wiki.parquet
• Training field: text

Architecture

Standard GPT-2-small architecture:

• Layers: 12
• Hidden size: 768
• Attention heads: 12
• Maximum context: 1024
• Vocabulary size: 43846

All GPT-2 weights were initialized from scratch.

No pretrained GPT-2 model weights were used.

Tokenizer

A separate Byte-Level BPE tokenizer was trained from scratch using only the Nepali training split.

The original English GPT-2 vocabulary and BPE merge rules were not used.

Vocabulary size: 43846

Training

• GPU: NVIDIA RTX A6000
• Precision: FP16
• TF32: True
• Maximum epochs: 50
• Batch size per GPU: 8
• Gradient accumulation: 4
• Sequence length: 1024
• Learning rate: 5e-05
• Scheduler: linear
• Validation frequency: every 500 optimizer steps
• Early stopping patience: 999
• Early stopping threshold: 0.0
• Random seed: 42

Training was stopped after validation loss failed to improve for 999 consecutive evaluation rounds, unless the maximum epoch ceiling was reached first.

Best Model

Best validation loss:

1.5498346090316772

Final best-model validation loss:

1.5498346090316772

Perplexity:

4.71069101241065