CAPIVARA

Model Card for CAPIVARA

CAPIVARA is a cost-efficient framework designed to enhance the performance of multilingual CLIP models in low-resource languages. This model holds the state of the art in many zero-shot tasks involving images and Portuguese texts.

How to use

import open_clip

model, preprocess_train, preprocess_val = open_clip.create_model_and_transforms('hf-hub:hiaac-nlp/CAPIVARA')
tokenizer = open_clip.get_tokenizer('hf-hub:hiaac-nlp/CAPIVARA')

For more details refer to Github repo.

Model Details

Model Description

CAPIVARA is built upon pre-trained OpenCLIP ViT-B/32 XLM-Roberta Base and fine-tuned with Conceptual Captions and synthetic captions generated by BLIP2. All the captions are translated with Google Translator.

Uses

Direct Use

Zero-shot image classification, zero-shot image and text retrieval, etc.

Downstream Use

Image classification and other image task fine-tuning, linear probe image classification, image captioning, image generation guiding and conditioning, etc.

Ethical considerations

For ethical considerations, please, refer to the Model Cards section in the paper.

Training Details

Training Data

The model was fine-tuned with Conceptual Captions and synthetic captions generated by BLIP2. All the captions are translated with Google Translator.

Training Hyperparameters

Optimizer: "Adam"
eps: 1e-8
weight_decay: 0.2
betas: [ 0.9, 0.98 ]

LR_scheduler: "CosineWarmupLR"
min_learning_rate: 1e-7
max_learning_rate: 5e-7
warmup_lr: 500

batch_size: 2816
max_steps: 5863 # 10 epochs

Evaluation

• Zero-shot image classification
• Zero-shot cross-modal retrieval

Testing Data, Factors & Metrics

Testing Data

For cross-modal retrieval, we used PraCegoVer, which is composed of images annotated originally with Portuguese texts, and our Portuguese-translated versions of MS COCO and Flickr30k. We also translate the labels from ImageNet and the
ELEVATER benchmark datasets for image classification.

Results

Zero-shot Cross-Modal Retrieval

We conducted zero-shot cross-modal retrieval experiments on Flickr30k and MS COCO with captions translated into Portuguese, and PraCegoVer. We report the average and standard deviation for 3 runs.

<table> <thead> <tr> <th>Models</th> <th colspan="2">Flickr30k</th> <th colspan="2"> MS COCO</th> <th colspan="2">PraCegoVer</th> </tr> </thead> <tbody> <tr> <td></td> <td>text-to-image</td> <td> image-to-text</td> <td>text-to-image</td> <td> image-to-text</td> <td>text-to-image</td> <td> image-to-text</td> </tr> <tr> <td>OpenCLIP ViT-B/32 XLM-Roberta Base (Baseline)</td> <td>76.23</td> <td>87.93</td> <td>52.62</td> <td>66.55</td> <td>65.36</td> <td><b>69.43</b></td> </tr> <tr> <td>CAPIVARA</td> <td><b>79.56 ± 0.01</b></td> <td><b>89.95 ± 0.04</b></td> <td><b>56.27 ± 0.01</b></td> <td><b>71.24 ± 0.01</b></td> <td><b>66.40 ± 0.01</b></td> <td>64.75 ± 0.01</td> </tr> </tbody> </table>

Zero-shot image classification

Environmental Impact

• GPU: 1 x Quadro RTX 8000 (48 GB)
• Hours used: 31 hours
• Compute Region: Brazil
• Carbon footprint: 0.5 Kg
• Energy: 6.49 kW

Citation

@inproceedings{santos2023capivara,
  title={CAPIVARA: Cost-Efficient Approach for Improving Multilingual CLIP Performance on Low-Resource Languages},
  author={Santos, Gabriel O. dos and Moreira, Diego A. B. and Ferreira, Alef I. and Silva, Jhessica and Pereira, Luiz and Bueno, Pedro and Sousa, Thiago and Maia, Helena and da Silva, N{\'a}dia and Colombini, Esther and Pedrini, Helio and Avila, Sandra},
  booktitle = "Workshop on Multi-lingual Representation Learning (MRL), Conference on Empirical Methods in Natural Language Processing (EMNLP)",
  year = "2023"
}