babylm-multilingual-gpt2-baseline

BabyLM Trilingual GPT-2 Baseline

This is a 97.7M-parameter GPT-2 causal language model trained from scratch on English, Dutch, and Chinese BabyBabel data for the BabyLM multilingual track. The three languages use equal sampling weights.

Model details

• Architecture: GPT2LMHeadModel
• Parameters: 97,737,216
• Layers: 12
• Attention heads: 12
• Hidden size: 768
• Context length: 512 tokens
• Tokenizer: byte-level BPE, 16,000 tokens
• Objective: causal language modeling
• Training epochs: 10
• Seed: 1

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "koki324/babylm-multilingual-gpt2-baseline"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

inputs = tokenizer("Hello world", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Training configuration

• English-equivalent token budget: 100,000,000
• Language weights: English 0.3333, Dutch 0.3333, Chinese 0.3333
• Sequence length: 512
• Per-device batch size: 32
• Gradient accumulation steps: 8
• Peak learning rate: 6e-4
• Warmup steps: 500
• BF16 training: enabled

Evaluation

The full evaluation output is included in evaluation_results.json. Selected zero-shot results include:

Intermediate checkpoints

Inference-ready intermediate checkpoints are provided under checkpoints/ at steps 759, 1518, 2277, 3036, 3795, 4554, 5313, 6072, 6831, and 7590. Optimizer, scheduler, and RNG states are intentionally excluded from this public package.

Limitations

This is a compact research baseline, not an instruction-tuned or safety-tuned assistant. Generated text may be inaccurate, biased, offensive, or repetitive. Do not use it for high-stakes decisions. Performance outside English, Dutch, and Chinese has not been established.

License

No model license has been selected yet. The repository owner must verify the training-data terms and choose an appropriate license before publication.