deberta-base-75k-sam_ext-s1

deberta-base-75k-sam_ext-s1 — BabyLM 2026 strict-small

DeBERTa-v3-base, identical recipe to deberta-base-75k-sam, with extended vision initialization: in addition to 21,134 tokens seeded from real grounding data (Flickr30k Entities, RefCOCO/g/+, THINGS; SAM ViT-B features), 737 further tokens covering 1,151 concrete zero-support words were seeded via a synthetic pipeline (LLM-written scene descriptions → SDXL-Turbo images ×3 → OWLv2 open-vocabulary detection → SAM features pooled in detected boxes, through the same extraction code). Total seeded: 21,871/75,000. The synthetic pipeline affects embedding initialization only — no synthetic images or descriptions enter the training text.

Training data: custom ~9.9M-word corpus (bb24.train) from our BabyLM 2024 submission (Edman et al. 2024, "Are BabyLMs Second Language Learners?") — a mixture of LLM-synthesized paraphrase/contrastive data (SynCSE-partial; Zhang et al. 2021) and portions of the official BabyLM corpus (Simple Wikipedia, Gutenberg, Switchboard). Within the strict-small 10M-word budget.

On a purpose-built visual-property benchmark (VP-Swap), the extension keeps the seeding gain over the text-only baseline but does not improve on the non-extended model across 3 training seeds (whole benchmark: +0.5 / 0.0 / −1.0 points; COMPS: +0.11 / +0.11 / −0.37).

Intermediate checkpoints: chck_1M … chck_100M, step1000 … step25740; main = final. Code, benchmark, analyses: https://github.com/bylinina/augustinian_babylm