clip-asymmetry

CLIP Asymmetric Encoder-Scaling Models (CC12M)

A grid of 30 CLIP models trained on CC12M that vary the sizes of the vision and text encoders independently. They accompany the paper Bigger Text Encoders Can Hurt CLIP Zero-Shot Performance and are meant for studying how the capacity split between the two encoders affects downstream performance.

Model Details

• Developed by: Samir Char, Carles Domingo-Enrich, Randall Balestriero
• Model type: Contrastive Language-Image Pretraining (CLIP); dual-encoder (ViT image encoder + Transformer text encoder)
• Language: English
• License: MIT
• Paper: Bigger Text Encoders Can Hurt CLIP Zero-Shot Performance, arXiv:2609.05730
• Code: https://github.com/samirchar/clip-asymmetry

The release is a 5 x 6 grid: vision encoders {ViT-Tiny-16, ViT-Atto-16, ViT-B-16, ViT-Giant-16, ViT-Colossal-16} crossed with text encoders {Femto, Nano, Atto, Tiny, Base, Giant}. Each model lives in a folder named {VisionEncoder}--{TextEncoder} containing a params.txt and checkpoints/epoch_latest.pt.

Uses

Direct use. Zero-shot image classification and image-text retrieval, and research on encoder scaling / modality-specific capacity in CLIP.

Out-of-scope use. These are research models trained on web-scraped CC12M; they are not production-ready and should not be used for high-stakes decisions. They can reflect biases present in the training data.

How to Use

These are custom OpenCLIP architectures (each a {Vision}--{Text} pairing that is not part of stock OpenCLIP), so first install the code repo, which registers them:

git clone https://github.com/samirchar/clip-asymmetry
cd clip-asymmetry
pip install -e .

The checkpoint is then pulled from this Hub repo automatically with hf_hub_download, so no manual download is required:

from huggingface_hub import hf_hub_download
from src.utils import register_models
import open_clip

register_models()  # adds the custom {Vision}--{Text} architectures to OpenCLIP's registry

ckpt = hf_hub_download(
    repo_id="samirchar/clip-asymmetry",
    filename="ViT-B-16--Base/checkpoints/epoch_latest.pt",
)
model, _, preprocess = open_clip.create_model_and_transforms("ViT-B-16--Base", pretrained=ckpt)
tokenizer = open_clip.get_tokenizer("ViT-B-16--Base")

Swap ViT-B-16--Base for any model in the grid.

Training Details

• Data: CC12M (pixparse/cc12m-wds), original captions.
• Framework: OpenCLIP.
• Recipe: 35 epochs, batch size 256, AdamW, learning rate 1e-3, weight decay 0.5, 1339 warmup steps, mixed precision (amp), image size 224.

Evaluation

Evaluated zero-shot and with linear probes on the clip-benchmark suite (ImageNet and its distribution shifts, MSCOCO / Flickr30k retrieval, VTAB, and others). See the paper for full results.

Citation

@article{char_clip_asymmetry,
  title         = {Bigger Text Encoders Can Hurt CLIP Zero-Shot Performance},
  author        = {Char, Samir and Domingo-Enrich, Carles and Balestriero, Randall},
  year          = {2026},
  eprint        = {2609.05730},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  doi           = {10.48550/arXiv.2609.05730}
}