Downloads · 30 days
19
50% of all-time downloads
babylm-roformer
bean4259/babylm-roformer
babylm-roformer is a fill-mask model from bean4259. Use it when you need the model to fill a missing word. The card lists the license as mit.
A BERT-style masked language model trained from scratch on the BabyLM 10M dataset.
All-time downloads
38
Public
Parameters
10.6M
42.4 MB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors42.4 MB · 99%
At a glance
- Task
- Fill-Mask
- License
- mit
- Model type
- roformer
- Access
- Public
- Created
- Jan 9, 2026
- Updated
- Jan 9, 2026
- SHA
- 7aa57ae1
Try a prompt
Model card
From the Hugging Face model README
BabyLM RoFormer (10M tokens)
A BERT-style masked language model trained from scratch on the BabyLM 10M dataset.
Model Details
- Architecture: RoFormer (BERT + Rotary Position Embeddings)
- Parameters: ~10M
- Training Data: BabyLM Strict-Small (10M tokens)
- Vocabulary: 16,384 tokens (WordPiece)
- Context Length: 128 tokens
Usage
from transformers import RoFormerForMaskedLM, RoFormerTokenizer
model = RoFormerForMaskedLM.from_pretrained("bean4259/babylm-roformer")
tokenizer = RoFormerTokenizer.from_pretrained("bean4259/babylm-roformer")
# Fill-mask example
text = "The cat sat on the [MASK]."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)
Training
Trained using a custom training loop with:
- Sequence packing (8.26x compression)
- AdamW optimizer (lr=1e-4)
- Linear warmup + decay schedule
- 10 epochs
- Task
- Fill-Mask
- Type
- roformer
- License
- mit
- Languages
- en
- Created
- Jan 9, 2026
- Updated
- Jan 9, 2026