morpheus-10M-v2

morpheus-10M-v2 — BabyLM 2026 Strict-Small (10M) submission

A 150M-parameter Llama-architecture causal LM trained from scratch on the BabyLM 10M-word strict-small corpus, using a morphological tokenizer (MorphyNet + hand-curated analyses; every token is a linguistic morpheme, ▁ marks word-initial morphemes, text is lowercased, OOV words map to <unk>).

• Architecture: LlamaForCausalLM, hidden 768, 12 layers, 12 heads, intermediate 3072, ctx 512, RoPE, tied embeddings
• Vocab: 48,102 morpheme tokens (our_vocab.json)
• Training: 10 epochs, AdamW (lr 6e-4, betas 0.9/0.95, wd 0.1), cosine schedule 2% warmup, bf16
• This checkpoint: best validation loss (3.4105577234788376 at step 1000, epoch 4.62962962962963)
• Intermediate checkpoints: branches chck_1M ... chck_100M (words seen), per BabyLM checkpoint requirements

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("juand-r/morpheus-10M-v2", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("juand-r/morpheus-10M-v2")

The tokenizer returns word-granularity offset_mapping, as required by the BabyLM strict evaluation harness.

Tokenizer provenance

Analyzer resources are the exact snapshot used to build the training data (repo commit 527db5f, verified to reproduce the training token stream exactly). char_backoff=False matches training (OOV -> <unk>); the tokenizer class also supports char_backoff=True (OOV -> character pieces).