CLIP-UniRepLKNet-L-laion5B-s10B-b75k

Model card for CLIP-UniRepLKNet_large-laion5B-s11B-b75K

Table of Contents

1. Quick Start
2. Model Details
3. Uses
4. Training Details
5. Evaluation
6. Citation

Quick Start

model_large = unireplknet_l() ## provided in modeling_UniRepLKNet.py
print(model_large)
ckpt = torch.load("UniRepLKNet-L-b75k_s10B_CLIP-in1k_75.72.pt")
model_large.load_state_dict(ckpt,strict=False) 
# Since we do not need cls heads in CLIP pretraining.
print("Loaded CLIP Pretrained Models")

Model Details

Model Description

A series of CLIP UniRepLKNet models trained on LAION-2B (english), a subset of LAION-5B, using OpenCLIP.

The core training run was performed in pieces over a period of ~ 2 months. The global batch size for the core run was 76800. The last ~25% of training was re-done at a 320x320 image resolution. See more details in Training Details.

Goals:

• Push the size of largest convolutional CLIP image tower into the performance range of ViT-g to ViT-G w/ improved image size scaling for downstream use.

Firsts:

• Largest released ConvNeXt model pretrained (847M params w/ 198 GMAC and 125 MActs @ 256x256 for image)
• A non-ViT image tower CLIP model (with no previous image tower pretrain) achieving > 79% ImageNet top-1 zero-shot

The models utilize:

• the UniRepLKNet model (unireplknet_l) as the image tower
• a standard projection at end of image tower
• a text tower with same size (with 1024, heads 16, depth 24) as ViT-H-14 and ViT-g-14 models

The models are trained at 224x224 image resolution at the beginning 192 epochs, then they're rewinded to 320x320 image resolution in the following 64 epochs. The size of the combined image + text CLIP model is 1.2B params w/ 222 GMAC and 146 MActs. UniRepLKNet excels in the both training and inference efficiency.

Model training done by Ross Wightman across both the stability.ai cluster and the JUWELS Booster supercomputer. See acknowledgements below.

Uses

As per the original OpenAI CLIP model card, this model is intended as a research output for research communities. We hope that this model will enable researchers to better understand and explore zero-shot, arbitrary image classification. We also hope it can be used for interdisciplinary studies of the potential impact of such model.

The OpenAI CLIP paper includes a discussion of potential downstream impacts to provide an example for this sort of analysis. Additionally, the LAION-5B blog (https://laion.ai/blog/laion-5b/) and upcoming paper include additional discussion as it relates specifically to the training dataset.

Direct Use

Zero-shot image classification, image and text retrieval, among others.

Downstream Use

Image classification and other image task fine-tuning, linear probe image classification, image generation guiding and conditioning, among others.

Out-of-Scope Use

As per the OpenAI models,

Any deployed use case of the model - whether commercial or not - is currently out of scope. Non-deployed use cases such as image search in a constrained environment, are also not recommended unless there is thorough in-domain testing of the model with a specific, fixed class taxonomy. This is because our safety assessment demonstrated a high need for task specific testing especially given the variability of CLIP’s performance with different class taxonomies. This makes untested and unconstrained deployment of the model in any use case currently potentially harmful.

Certain use cases which would fall under the domain of surveillance and facial recognition are always out-of-scope regardless of performance of the model. This is because the use of artificial intelligence for tasks such as these can be premature currently given the lack of testing norms and checks to ensure its fair use.

Since the model has not been purposefully trained in or evaluated on any languages other than English, its use should be limited to English language use cases.

Further the above notice, the LAION-5B dataset used in training of these models has additional considerations, see below.

Training Details

Training Data

This model was trained with LAION-2B -- A 2 billion sample English subset of LAION-5B (https://laion.ai/blog/laion-5b/).

Training Procedure

The main training run was done at global batch size of 76800 for 256 checkpoint intervals of 45M samples for a total of ~11.5B samples seen over training.

A slurm srun command line below for a 128 8-GPU (40GB A100) configuration:

srun --cpu_bind=v --accel-bind=gn python -m training.main \
    --save-frequency 1 \
    --name "large-5b-76800-bf16" \
    --resume "latest" \
    --logs "/runs" \
    --log-every-n-steps 100 \
    --train-data="pipe:aws s3 cp s3://laion5b/laion2B-data/{000000..231349}.tar -" \
    --train-num-samples 45646078 \
    --dataset-type webdataset \
    --warmup 10000 \
    --batch-size=600 \
    --epochs=256 \
    --dataset-resampled \
    --precision amp_bfloat16 \
    --grad-clip-norm 5.0 \
    --lr 5e-4 \
    --workers=6 \
    --beta2 0.99 \
    --model "unireplknet_l" \
    --seed 0 \
    --ddp-static-graph \
    --local-loss \
    --gather-with-grad \
    --grad-checkpointing \
    --report-to "tensorboard"

Evaluation

Evaluation done with code in the LAION CLIP Benchmark suite.

Results

These models achieve between 75.7 and 76.3 top-1 zero-shot accuracy on ImageNet-1k.

Citation

BibTeX:

UniRepLKNet

@inproceedings{ding2024unireplknet,
  title={UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio Video Point Cloud Time-Series and Image Recognition},
  author={Ding, Xiaohan and Zhang, Yiyuan and Ge, Yixiao and Zhao, Sijie and Song, Lin and Yue, Xiangyu and Shan, Ying},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={5513--5524},
  year={2024}
}

LAION-5B

@inproceedings{schuhmann2022laionb,
  title={{LAION}-5B: An open large-scale dataset for training next generation image-text models},
  author={Christoph Schuhmann and
          Romain Beaumont and
          Richard Vencu and
          Cade W Gordon and
          Ross Wightman and
          Mehdi Cherti and
          Theo Coombes and
          Aarush Katta and
          Clayton Mullis and
          Mitchell Wortsman and
          Patrick Schramowski and
          Srivatsa R Kundurthy and
          Katherine Crowson and
          Ludwig Schmidt and
          Robert Kaczmarczyk and
          Jenia Jitsev},
  booktitle={Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
  year={2022},
  url={https://openreview.net/forum?id=M3Y74vmsMcY}
}

OpenCLIP software

@software{ilharco_gabriel_2021_5143773,
  author       = {Ilharco, Gabriel and
                  Wortsman, Mitchell and
                  Wightman, Ross and
                  Gordon, Cade and
                  Carlini, Nicholas and
                  Taori, Rohan and
                  Dave, Achal and
                  Shankar, Vaishaal and
                  Namkoong, Hongseok and
                  Miller, John and
                  Hajishirzi, Hannaneh and
                  Farhadi, Ali and
                  Schmidt, Ludwig},
  title        = {OpenCLIP},
  month        = jul,
  year         = 2021,
  note         = {If you use this software, please cite it as below.},
  publisher    = {Zenodo},
  version      = {0.1},
  doi          = {10.5281/zenodo.5143773},
  url          = {https://doi.org/10.5281/zenodo.5143773}
}

OpenAI CLIP paper

@misc{rw2019timm,
  author = {Ross Wightman},
  title = {PyTorch Image Models},
  year = {2019},
  publisher = {GitHub},
  journal = {GitHub repository},
  doi = {10.5281/zenodo.4414861},
  howpublished = {\url{https://github.com/rwightman/pytorch-image-models}}
}