pathlang-1p7b-runB-en-first

pathlang-1p7b-runB-en-first

A 1.7B-parameter bilingual (English + Chinese) language model, part of a controlled language-ordering study. Three models share the same architecture, initialization, total data diet (50B English + 50B Chinese = 100B tokens), and LR schedule; they differ only in the order in which the two languages are presented during pretraining. This is the English-first run.

Curriculum (this run)

• Phase 1 (0-30B): 100% English
• Phase 2 (30-90B): 75% Chinese / 25% English
• Phase 3 (90-100B): 50/50

The other runs in the study: Raghav-Singhal/pathlang-1p7b-runA-zh-first, Raghav-Singhal/pathlang-1p7b-runB-en-first, Raghav-Singhal/pathlang-1p7b-runC-5050.

Architecture

• SmolLM2-1.7B backbone: 24 layers, hidden size 2048, FFN 8192, 32 attention heads
• RoPE (base 10000), RMSNorm, SwiGLU, no biases, sequence length 2048
• Tokenizer: Qwen3 (multilingual, vocab 151,936)
• ~2.1B total parameters (with the Qwen3 embedding)

Training

• 100B tokens: 50B English (DCLM-edu) + 50B Chinese (FineWeb-2 cmn_Hani)
• Global batch size 960, sequence length 2048, 50,860 steps
• WSD LR schedule (peak 2e-4, 2000 warmup, linear decay over the final 10B tokens), bf16, Adam
• Converted from Megatron-LM to HF LlamaForCausalLM format

Held-out validation loss (final checkpoint, 100B tokens)

Cross-entropy in nats/token on held-out blocks of the training corpora. The headline finding of the study: each run ends best at the language that dominated its middle (bulk) phase — a recency effect that also holds on an out-of-distribution corpus (HPLT4.0).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("Raghav-Singhal/pathlang-1p7b-runB-en-first")
tok = AutoTokenizer.from_pretrained("Raghav-Singhal/pathlang-1p7b-runB-en-first")

Note

License is set to other pending confirmation; the training data (DCLM-edu, FineWeb-2) and their respective terms apply. This is a base (non-instruction-tuned) research checkpoint.