deberta-base-75k-sam

deberta-base-75k-sam — BabyLM 2026 strict-small

DeBERTa-v3-base trained from scratch (10 epochs, ≤100M words exposure), 75k byte-level BPE tokenizer. Vision-initialized: before training, input embeddings of the 21,134 vocabulary tokens attested in visual grounding data (Flickr30k Entities, RefCOCO/g/+, THINGS) were seeded from SAM ViT-B region features (bbox-patch pooling, mean-centered, L2-normalized, scaled); all other tokens use standard random init. Training: lr 2e-4, batch 256, grad-acc 4, MLM 15%.

Training data: custom ~9.9M-word corpus (bb24.train) from our BabyLM 2024 submission (Edman et al. 2024, "Are BabyLMs Second Language Learners?") — a mixture of LLM-synthesized paraphrase/contrastive data (SynCSE-partial; Zhang et al. 2021) and portions of the official BabyLM corpus (Simple Wikipedia, Gutenberg, Switchboard). Within the strict-small 10M-word budget.

Intermediate checkpoints: revisions chck_1M … chck_100M (words seen) and step1000 … step25740; main = final.

Findings (vs the matched control deberta-base-75k): consistent gains on object property knowledge (COMPS; and a purpose-built visual-property benchmark, +1.9–3.0 pts across 3 seeds), no effect on syntax. Code, benchmark, analyses: https://github.com/bylinina/augustinian_babylm