Persian-CLIP

<!-- This model card has been generated automatically according to the information the Trainer had access to. You should probably proofread and complete it, then remove this comment. -->

persian-clip

This model is a fine-tuned version of on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.7629

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 3e-05
• train_batch_size: 32
• eval_batch_size: 32
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 100
• num_epochs: 5

Training results

Framework versions

• Transformers 4.38.2
• Pytorch 2.1.2+cu121
• Datasets 2.10.1
• Tokenizers 0.15.0

How to use?

# Both models generate vectors with 768 dimensions.
from transformers import CLIPVisionModel, RobertaModel, AutoTokenizer, CLIPFeatureExtractor
# download pre-trained models
vision_encoder = CLIPVisionModel.from_pretrained('SeyedAli/Persian-CLIP')
preprocessor = CLIPFeatureExtractor.from_pretrained('SeyedAli/Persian-CLIP')
text_encoder = RobertaModel.from_pretrained('SeyedAli/Persian-CLIP')
tokenizer = AutoTokenizer.from_pretrained('SeyedAli/Persian-CLIP')
# define input image and input text
text = 'something'
image = PIL.Image.open('my_favorite_image.jpg')
# compute embeddings
text_embedding = text_encoder(**tokenizer(text,
                                          return_tensors='pt')).pooler_output
image_embedding = vision_encoder(**preprocessor(image, 
                                                return_tensors='pt')).pooler_output

zero-shot-Image-Classification:

The followings are just some use cases of Persian-CLIP on 25K Unsplash images

• use pip install -q git+https://github.com/sajjjadayobi/clipfa.git

from clipfa import CLIPDemo
import torch
# Both models generate vectors with 768 dimensions.
from transformers import CLIPVisionModel, RobertaModel, AutoTokenizer, CLIPFeatureExtractor
# download pre-trained models
vision_encoder = CLIPVisionModel.from_pretrained('SeyedAli/Persian-CLIP')
preprocessor = CLIPFeatureExtractor.from_pretrained('SeyedAli/Persian-CLIP')
text_encoder = RobertaModel.from_pretrained('SeyedAli/Persian-CLIP')
tokenizer = AutoTokenizer.from_pretrained('SeyedAli/Persian-CLIP')

demo = CLIPDemo(vision_encoder, text_encoder, tokenizer)
demo.compute_text_embeddings(['متن 3' ,'متن 2' ,'متن 1'])
demo.compute_image_embeddings(['my_favorite_image.jpg'])
demo.zero_shot(image_path='my_favorite_image.jpg')