roberta-clip-cc12m-100pct

roberta-clip-cc12m-100pct

RoBERTa-base contrastively aligned with CC12M image-caption pairs, following a CLIP-style recipe with target-word markers. Released as part of a master thesis investigating whether multimodal contrastive pretraining improves semantic-proximity / Word-in-Context (WiC) classification.

Model details

• Text encoder: roberta-base, all 12 layers unfrozen
• Image tower (frozen): ViT-L-14 (OpenCLIP, openai weights)
• Projection: linear, 768-d, learned separately for image and text features
• Loss: symmetric InfoNCE, logit scale clamped to [log(1/100), log(100)], initialised at log(1/0.07); DDP ranks all-gather features so every other rank contributes as negatives
• Target-word markers: each caption is wrapped with <t>...</t> around a randomly chosen word

Training

• Data: pixparse/cc12m-wds (CC12M, webdataset shards)
• Steps: 100,000
• Per-device batch size: 128, gradient accumulation 1, max grad norm 1.0
• Optimizer: AdamW, weight decay 0.01
• Schedule: linear warmup 1,000 steps
• LRs: encoder 2e-5 with per-layer decay 0.85 from the top; image/text projection heads2e-4; logit_scale 2e-4, weight decay 0
• Precision: fp32