ViCLIP-OT

<div align="center"> <h3 align="center"> <b> ViCLIP-OT &mdash; The First Foundation Vision-Language Model for Vietnamese Image–Text Retrieval with Optimal Transport </b> </h3> </div> <div align="center"> <a href="https://arxiv.org/abs/2602.22678"><img alt="arXiv" src="https://img.shields.io/badge/arXiv-2602.22678-b31b1b.svg?style=flat"></a> <a href="https://github.com/minhnguyent546/ViCLIP-OT"><img alt="Github repository" src="https://img.shields.io/badge/GitHub-100000?style=flat&logo=github&logoColor=white"></a> <a href="https://huggingface.co/collections/minhnguyent546/viclip-ot"><img alt="Model on Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Model-blue"></a> </div>

ViCLIP-OT is the first foundation vision-language model specifically designed for Vietnamese image-text retrieval.

ViCLIP-OT models combine CLIP-style contrastive learning with a novel Similarity-Graph Regularized Optimal Transport (SIGROT) loss to enhance cross-modal alignment and reduce the modality gap. ViCLIP-OT achieves state-of-the-art performance on Vietnamese image-text retrieval benchmarks with strong zero-shot generalization.

Model Variants

Quick Start

Model Overview

ViCLIP-OT is a dual-encoder vision-language model with 221M parameters. The text encoder is based on Vietnamese-SBERT, and the image encoder uses a ViT-B/16 backbone pre-trained with the DINOv3 framework. The model is trained with a hybrid objective that combines CLIP-style contrastive learning and the proposed Similarity-Graph Regularized Optimal Transport (SIGROT) loss.

Intended Uses

ViCLIP-OT is a multimodal embedding model that encodes both Vietnamese text and images into a shared representation space. It can be used for:

• Vietnamese image-text retrieval (text-to-image and image-to-text, image-to-image)
• Cross-modal semantic search in Vietnamese
• Feature extraction for downstream Vietnamese vision-language tasks (e.g., visual question answering, image classification)

Usage

<details> <summary>via <a href="https://huggingface.co/docs/transformers/en/index">transformers</a></summary>

pip install \
    'transformers>=4.57.0,<5.0.0' \
    'torch>=2.8.0,<2.10.0' \
    'torchvision>=0.23.0,<0.25.0' \
    timm \
    pillow

from transformers import AutoModel, AutoProcessor
import torch

# Initialize the model
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(f'Using device: {device}')
model = AutoModel.from_pretrained('minhnguyent546/ViCLIP-OT', trust_remote_code=True)
model.to(device)

# Example images and sentences
image_uris = [
    'http://images.cocodataset.org/train2014/COCO_train2014_000000138621.jpg',
    'http://images.cocodataset.org/train2014/COCO_train2014_000000190580.jpg',
]
sentences = [
    'Một con mèo màu trắng',
    'Một con mèo màu đen',
    'Một cô gái đang lướt sóng',
]

# Encode text
text_embeddings = model.encode_text(
    sentences=sentences,
    batch_size=32,
    show_progress_bar=True,
    convert_to_tensor=True,
    normalize=True,
    padding=True,
    truncation=True,
    max_length=512,
)

# Encode images
image_embeddings = model.encode_image(
    images=image_uris,
    batch_size=32,
    show_progress_bar=True,
    convert_to_tensor=True,
    normalize=True,
)

# Compute cosine similarity between image and text embeddings
similarities = image_embeddings @ text_embeddings.T
print(similarities)

# tensor([[0.2438, 0.1506, 0.7248],
#         [0.4299, 0.5287, 0.2329]])

</details> <details> <summary>via <a href="https://sbert.net/">sentence-transformers</a></summary>

pip install \
    'transformers>=4.57.0,<5.0.0' \
    'sentence-transformers>=4.0.0' \
    'torch>=2.8.0,<2.10.0' \
    'torchvision>=0.23.0,<0.25.0' \
    timm \
    pillow

from sentence_transformers import SentenceTransformer

# Initialize the model
model = SentenceTransformer('minhnguyent546/ViCLIP-OT', trust_remote_code=True)

# Example images and sentences
image_uris = [
    'http://images.cocodataset.org/train2014/COCO_train2014_000000138621.jpg',
    'http://images.cocodataset.org/train2014/COCO_train2014_000000190580.jpg',
]
sentences = [
    'Một con mèo màu trắng',
    'Một con mèo màu đen',
    'Một cô gái đang lướt sóng',
]

# Encode text
text_embeddings = model.encode(
    sentences,
    batch_size=32,
    show_progress_bar=True,
    convert_to_tensor=True,
    normalize_embeddings=True,
)

# Encode images
image_embeddings = model.encode(
    image_uris,
    batch_size=32,
    show_progress_bar=True,
    convert_to_tensor=True,
    normalize_embeddings=True,
)

# Compute cosine similarity between image and text embeddings
similarities = image_embeddings @ text_embeddings.T
print(similarities)

# tensor([[0.2438, 0.1506, 0.7248],
#         [0.4299, 0.5287, 0.2329]])

</details>

Training Details

ViCLIP-OT is trained on UIT-OpenViIC, a large-scale open-domain Vietnamese image captioning dataset containing 9,088 training images with approximately 42,000 captions featuring diverse real-world scenes.

For more details, please refer to the GitHub repository.

Evaluation Results

Image-Text Retrieval on UIT-OpenViIC

The table below summarizes retrieval performance on the UIT-OpenViIC test set. Both models also substantially outperform pretrained multilingual vision-language models evaluated in a zero-shot setting.

<table> <caption> <strong>Table:</strong> Image-text retrieval performance on the test set of the UIT-OpenViIC dataset. UOT denotes Unbalanced Optimal Transport. * indicates zero-shot evaluation. Best results are in bold and second-best are underlined. </caption> <thead> <tr> <th rowspan="2">Method/Model</th> <th rowspan="2"># Params</th> <th colspan="3">Text → Image</th> <th colspan="3">Image → Text</th> <th rowspan="2">Avg.</th> </tr> <tr> <th>R@1</th> <th>R@5</th> <th>R@10</th> <th>R@1</th> <th>R@5</th> <th>R@10</th> </tr> </thead> <tbody> <tr> <td>mSigLIP-base*</td> <td>370M</td> <td>14.34</td> <td>28.94</td> <td>36.21</td> <td>20.49</td> <td>32.23</td> <td>37.43</td> <td>28.27</td> </tr> <tr> <td>Jina CLIP v2*</td> <td>865M</td> <td>30.01</td> <td>52.09</td> <td>61.70</td> <td>40.23</td> <td>65.02</td> <td>74.41</td> <td>53.91</td> </tr> <tr> <td>Jina Embedding v4*</td> <td>4B</td> <td>23.97</td> <td>42.22</td> <td>50.29</td> <td>41.48</td> <td>66.77</td> <td>75.61</td> <td>50.06</td> </tr> <tr> <td>Qwen3-VL-Embedding-2B*</td> <td>2B</td> <td>32.13</td> <td>54.00</td> <td>62.93</td> <td>39.83</td> <td>66.52</td> <td>77.01</td> <td>55.40</td> </tr> <tr style="border-top: 2px solid #000;"> <td>CLIP</td> <td>221M</td> <td>31.19</td> <td>59.80</td> <td>71.23</td> <td>46.60</td> <td>75.53</td> <td>85.19</td> <td>61.59</td> </tr> <tr> <td>SigLIP</td> <td>221M</td> <td>34.75</td> <td>63.01</td> <td>72.96</td> <td>50.10</td> <td>79.78</td> <td>88.04</td> <td>64.77</td> </tr> <tr style="border-top: 2px solid #000;"> <td>CLIP + UOT</td> <td>221M</td> <td>29.27</td> <td>57.62</td> <td>69.07</td> <td>43.59</td> <td>75.03</td> <td>84.03</td> <td>59.77</td> </tr> <tr> <td>SigLIP + UOT</td> <td>221M</td> <td>37.84</td> <td>65.30</td> <td>74.98</td> <td>53.95</td> <td>80.95</td> <td>88.81</td> <td>66.97</td> </tr> <tr> <td>SIGROT</td> <td>221M</td> <td><strong>40.75</strong></td> <td><strong>70.72</strong></td> <td><strong>80.90</strong></td> <td>37.99</td> <td>61.11</td> <td>71.68</td> <td>60.53</td> </tr> <tr> <td><strong>ViCLIP-OT (Ours)</strong></td> <td>221M</td> <td>37.57</td> <td>65.65</td> <td>75.43</td> <td><u>54.35</u></td> <td><u>81.83</u></td> <td><u>89.19</u></td> <td><u>67.34</u></td> </tr> <tr> <td><strong>ViSigLIP-OT (Ours)</strong></td> <td>221M</td> <td><u>39.19</u></td> <td><u>66.71</u></td> <td><u>76.04</u></td> <td><strong>57.21</strong></td> <td><strong>83.83</strong></td> <td><strong>90.79</strong></td> <td><strong>68.96</strong></td> </tr> </tbody> </table>

Zero-shot image–text retrieval results on KTVIC and Crossmodal-3600

The table below reports zero-shot retrieval results on KTVIC (with near-duplicate images removed against the UIT-OpenViIC training set) and Crossmodal-3600 (using Vietnamese captions).

<table> <caption> <strong>Table:</strong> Zero-shot image–text retrieval results on KTVIC and Crossmodal-3600. KTVIC images are deduplicated against the UIT-OpenViIC training set. Vietnamese captions are used for Crossmodal-3600. </caption> <thead> <tr> <th rowspan="2">Method</th> <th colspan="3" style="text-align: center;">Text → Image</th> <th colspan="3" style="text-align: center;">Image → Text</th> <th rowspan="2">Avg.</th> </tr> <tr> <th>R@1</th> <th>R@5</th> <th>R@10</th> <th>R@1</th> <th>R@5</th> <th>R@10</th> </tr> </thead> <tbody> <tr> <td colspan="8" style="text-align: center; font-style: italic;"><strong>KTVIC – train</strong></td> </tr> <tr> <td>CLIP</td> <td>21.12</td> <td>46.99</td> <td>59.22</td> <td>31.65</td> <td>59.46</td> <td>72.49</td> <td>48.49</td> </tr> <tr> <td>SigLIP</td> <td>23.16</td> <td>48.78</td> <td>60.57</td> <td>35.48</td> <td>62.22</td> <td>73.64</td> <td>50.64</td> </tr> <tr> <td>ViCLIP-OT</td> <td>26.24</td> <td>52.46</td> <td>64.14</td> <td>38.47</td> <td>64.37</td> <td>75.48</td> <td>53.52</td> </tr> <tr> <td>ViSigLIP-OT</td> <td>26.28</td> <td>52.58</td> <td>63.49</td> <td>39.62</td> <td>66.44</td> <td>77.78</td> <td><strong>54.37</strong></td> </tr> <tr> <td colspan="8" style="text-align: center; font-style: italic;"><strong>KTVIC – test</strong></td> </tr> <tr> <td>CLIP</td> <td>50.32</td> <td>82.80</td> <td>89.94</td> <td>63.06</td> <td>92.36</td> <td>97.45</td> <td>79.32</td> </tr> <tr> <td>SigLIP</td> <td>52.61</td> <td>83.31</td> <td>89.94</td> <td>71.97</td> <td>94.27</td> <td>96.18</td> <td>81.38</td> </tr> <tr> <td>ViCLIP-OT</td> <td>56.69</td> <td>85.61</td> <td>91.97</td> <td>70.06</td> <td>93.63</td> <td>98.09</td> <td>82.68</td> </tr> <tr> <td>ViSigLIP-OT</td> <td>56.56</td> <td>85.99</td> <td>91.72</td> <td>71.34</td> <td>93.63</td> <td>97.45</td> <td><strong>82.78</strong></td> </tr> <tr> <td colspan="8" style="text-align: center; font-style: italic;"><strong>Crossmodal-3600</strong></td> </tr> <tr> <td>CLIP</td> <td>22.52</td> <td>45.55</td> <td>58.01</td> <td>26.22</td> <td>53.42</td> <td>65.06</td> <td>45.13</td> </tr> <tr> <td>SigLIP</td> <td>26.67</td> <td>50.31</td> <td>61.78</td> <td>31.17</td> <td>57.78</td> <td>69.83</td> <td>49.59</td> </tr> <tr> <td>ViCLIP-OT</td> <td>28.90</td> <td>55.29</td> <td>66.37</td> <td>42.56</td> <td>68.81</td> <td>79.17</td> <td><strong>56.85</strong></td> </tr> <tr> <td>ViSigLIP-OT</td> <td>32.04</td> <td>57.90</td> <td>68.95</td> <td>37.97</td> <td>64.64</td> <td>75.53</td> <td>56.17</td> </tr> </tbody> </table>

License

This project is licensed under the GNU General Public License v3.0 (GPL-3.0).

Citation

If you find ViCLIP-OT useful in your research, please cite the following paper:

@misc{tran2026viclipotfoundationvisionlanguagemodel,
  title={ViCLIP-OT: The First Foundation Vision-Language Model for Vietnamese Image-Text Retrieval with Optimal Transport}, 
  author={Quoc-Khang Tran and Minh-Thien Nguyen and Nguyen-Khang Pham},
  year={2026},
  eprint={2602.22678},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2602.22678}, 
}