deberta-base-50k

deberta-base-50k

Random-initialization baseline for the 50k vocabulary. This is the control that vision-seeded models with the same vocabulary are compared against.

Seeded counterparts: augustinian-babylm/deberta-base-50k-sam, -dinov3, -ibot.

Part of the model set for Augustinian BabyLM: What Ostensive Definition Can and Cannot Teach a Small Language Model (paper, code).

Usage

from transformers import AutoModelForMaskedLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("augustinian-babylm/babylm-bpe-50k")
model = AutoModelForMaskedLM.from_pretrained("augustinian-babylm/deberta-base-50k")

Training

DeBERTa-v3-base masked LM (768 hidden, 12 layers, 12 heads), trained for 10 epochs on bb24.train, the ~9.9M-word corpus of Edman et al. (2024), which mixes LLM-generated paraphrase data with portions of the official BabyLM corpus. This is not the official 2026 strict-small distribution.

Only the input-embedding initialization differs between a vision-init model and its baseline. Every other hyperparameter, the data order, and the random seed are held fixed.

Checkpoints

Intermediate checkpoints are stored as branches, saved Pythia-style by step:

from transformers import AutoModelForMaskedLM
AutoModelForMaskedLM.from_pretrained("augustinian-babylm/deberta-base-50k", revision="step256")

Call huggingface_hub.list_repo_refs("augustinian-babylm/deberta-base-50k") to list the available steps.

Results

Vision seeding produces a consistent object-property gain (COMPS, +1.30 averaged over nine encoder-by-vocabulary configurations, positive in all nine) while grammar-focused benchmarks stay flat. On a corpus-tailored Visual-Property Swap probe the seeded model leads the baseline in all three training seeds; a pair-level re-analysis shows that the advantage cannot be attributed to the individual seeded words. Details: https://github.com/bylinina/augustinian_babylm

Limitations

Trained on a non-standard corpus (see above), so results are not directly comparable to submissions trained on the official 2026 strict-small data. Single masked-LM architecture. Research artifact, not intended for deployment.

Citation

@inproceedings{bylinina2026augustinian,
  title     = {Augustinian BabyLM: What Ostensive Definition Can and Cannot
               Teach a Small Language Model},
  author    = {Bylinina, Lisa},
  booktitle = {Proceedings of the BabyLM Workshop},
  year      = {2026},
  url       = {https://openreview.net/forum?id=B4TD4XdlwF}
}