Downloads · 30 days
35
2% of all-time downloads
deberta-base-75k
augustinian-babylm/deberta-base-75k
deberta-base-75k is a machine learning model from augustinian-babylm. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
DeBERTa-v3-base trained from scratch (10 epochs, ≤100M words exposure), 75k byte-level BPE tokenizer, standard random embedding initialization — the matched control for deberta-base-75k-sam: identical data, architectu…
All-time downloads
1.8K
Public
Parameters
144M
20.7 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors575 MB · 99%
At a glance
- License
- apache-2.0
- Model type
- deberta-v2
- Access
- Public
- Created
- Jun 12, 2026
- Updated
- Jul 17, 2026
- SHA
- e5dbe1ec
Model card
From the Hugging Face model README
deberta-base-75k — BabyLM 2026 strict-small (random-init control)
DeBERTa-v3-base trained from scratch (10 epochs, ≤100M words exposure),
75k byte-level BPE tokenizer, standard random embedding initialization —
the matched control for
deberta-base-75k-sam:
identical data, architecture, tokenizer, and schedule; the only
difference is embedding initialization. Training: lr 2e-4, batch 256,
grad-acc 4, MLM 15%.
Training data: custom ~9.9M-word corpus (bb24.train) from our BabyLM
2024 submission (Edman et al. 2024, "Are BabyLMs Second Language
Learners?") — a mixture
of LLM-synthesized paraphrase/contrastive data (SynCSE-partial; Zhang et
al. 2021) and portions of the official BabyLM corpus (Simple Wikipedia,
Gutenberg, Switchboard). Within the strict-small 10M-word budget.
Intermediate checkpoints: chck_1M … chck_100M and step1000 …
step25740; main = final.
Code and analyses: https://github.com/bylinina/augustinian_babylm
- Type
- deberta-v2
- License
- apache-2.0
- Languages
- en
- Created
- Jun 12, 2026
- Updated
- Jul 17, 2026