gpt2-am-wikimulti

GPT-2 Amharic WikiMulti

GPT-2-small model pretrained from scratch on the Amharic portion of WikiMulti.

Dataset

• Dataset: dehanalkautsar/WikiMulti
• File: 20260801_am_wiki.parquet
• Training field: text

Architecture

Standard GPT-2-small architecture:

• Layers: 12
• Hidden size: 768
• Attention heads: 12
• Maximum context: 1024
• Vocabulary size: 50257

All GPT-2 weights were initialized from scratch.

No pretrained GPT-2 model weights were used.

Tokenizer

A separate Byte-Level BPE tokenizer was trained from scratch using only the Amharic training split.

The original English GPT-2 vocabulary and BPE merge rules were not used.

Vocabulary size: 50257

Training

• GPU: NVIDIA RTX A6000
• Precision: FP16
• TF32: True
• Maximum epochs: 50
• Batch size per GPU: 8
• Gradient accumulation: 4
• Sequence length: 1024
• Learning rate: 5e-05
• Scheduler: linear
• Validation frequency: every 500 optimizer steps
• Early stopping patience: 999
• Early stopping threshold: 0.0
• Random seed: 42

Training was stopped after validation loss failed to improve for 999 consecutive evaluation rounds, unless the maximum epoch ceiling was reached first.

Best Model

Best validation loss:

5.9864678382873535

Final best-model validation loss:

5.9864678382873535

Perplexity:

398.00630158504686