morpheus-10M-v3

morpheus-10M-v3 — BabyLM 2026 Strict-Small (10M) submission

150M-parameter Llama-architecture LM trained from scratch on the BabyLM 10M strict-small corpus with a morphological tokenizer (every token a linguistic morpheme; MorphyNet + hand-curated analyses; lowercased; ▁ marks word-initial morphemes; OOV words back off to character pieces).

Recipe (the overnight stack, BLiMP 70.45 on our strict reproduction of the 2025 pipeline, vs 68.0 for the v2 baseline model): hybrid causal + masked-next-token-prediction objective (50% of steps masked at 15%, bidirectional attention on masked batches, GPT-BERT-style output shift), context 256, attention dropout 0.1, AdamW lr 1e-3 (betas 0.9/0.95), weight decay 0.3, cosine schedule 2% warmup, 10 epochs (BabyLM cap), bf16.

• Vocab: 48,728 morpheme tokens (our_vocab.json), char-backoff pieces included
• Best-val checkpoint (step 4500, val 3.4322287875196964); milestone branches chck_1M..chck_100M
• The tokenizer loads with trust_remote_code=True, needs no extra pip deps, and returns word-granularity offset_mapping as the BabyLM strict harness requires.

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("juand-r/morpheus-10M-v3", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("juand-r/morpheus-10M-v3")

Tokenizer provenance: resources snapshot verified to reproduce the training token stream exactly (14,446/14,446 documents, 2026-07-20). Sibling model: juand-r/morpheus-10M-v2 (pure-causal baseline, ctx 512).