Downloads · 30 days
17
100% of all-time downloads
roberta-mlm-cc12m-100pct
NecroDevelopment/roberta-mlm-cc12m-100pct
roberta-mlm-cc12m-100pct is a machine learning model from NecroDevelopment. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
RoBERTa-base continually pre-trained on CC12M captions with masked language modeling under the same 100k-step, target-word-marker recipe used by the companion contrastive run. Ships as the text-only control in the WiC…
All-time downloads
17
Public
Repo size
496 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors496 MB · 99%
At a glance
- License
- mit
- Access
- Public
- Created
- Sep 10, 2026
- Updated
- Sep 10, 2026
- SHA
- e7f8a6df
Base models
Model card
From the Hugging Face model README
mlm-roberta-base-cc12m-100pct
RoBERTa-base continually pre-trained on CC12M captions with masked language modeling under the same 100k-step, target-word-marker recipe used by the companion contrastive run. Ships as the text-only control in the WiC semantic-proximity bake-off.
Model details
- Text encoder: roberta-base, all 12 layers unfrozen, MLM head trained on top
- Input: CC12M captions wrapped with
<t>...</t>around a randomly chosen word (mark_random_target=True,marker_min_word_len=3);target_pool=False - Objective: standard subword MLM, 15% masking probability
- Max sequence length: 77, set to match the contrastive run
Training
- Data:
pixparse/cc12m-wds(CC12M, webdataset shards), captions branch only - Steps: 100,000
- Per-device batch size: 128
- Optimizer: AdamW, weight decay 0.01
- Schedule: linear warmup 1,000 steps, then constant
- LRs: encoder 2e-5 with per-layer decay 0.85 from the top; MLM head 2e-4
- Precision: fp32
- License
- mit
- Languages
- en
- Created
- Sep 10, 2026
- Updated
- Sep 10, 2026