Downloads · 30 days
61
1% of all-time downloads
mlcd-vit-large-patch14-336
DeepGlint-AI/mlcd-vit-large-patch14-336
mlcd-vit-large-patch14-336 is a feature extraction model from DeepGlint-AI. Use it when you need embeddings to search or compare text. The card lists the license as apache-2.0.
[[Paper]](https://arxiv.org/abs/2407.17331) [[GitHub]](https://github.com/deepglint/unicom) We used the same Vision Transformer architecture ViT-L/14@336px as CLIP.
All-time downloads
6.7K
Public
Parameters
304M
1.2 GB on disk
Likes
10
Public
Repo makeup
Click a slice to open those files.
.safetensors1.2 GB · 100%
At a glance
- Task
- Feature Extraction
- License
- apache-2.0
- Model type
- clip_vision_model
- Access
- Public
- Created
- Oct 11, 2024
- Updated
- Nov 13, 2024
- SHA
- bd75627f
Model card
From the Hugging Face model README
Model
We used the same Vision Transformer architecture ViT-L/14@336px as CLIP.
Data
Our model was trained on publicly available image-caption data from the LAION400M and COYO700M datasets.
Performance and Limitations
A. MLLMs Evaluation Results
In our experiments, we replaced the CLIP model in LLaVA-NeXT with the MLCD model to demonstrate the performance of the MLCD model in Multimodal Large Language Models (MLLMs). For the language model, we used Qwen2.5-7B. The evaluation results show that the modified model performs exceptionally well across multiple benchmarks, validating the effectiveness of the MLCD model within MLLMs.
| Vision Tower | MLCD (ViT_L_14_336px) | CLIP (ViT_L_14_336px) |
|---|---|---|
| LLM | Qwen2.5-7B | Qwen2.5-7B |
| AI2D | <span style="color:red">76.98</span> | 73.15 |
| ScienceQA_img | <span style="color:red">78.09</span> | 76.35 |
| GQA | <span style="color:red">64.17</span> | 63.31 |
| InfoVQA_val | <span style="color:red">43.48</span> | 38.88 |
| MMBench_cn_dev | <span style="color:red">74.83</span> | 72.51 |
| MMBench_en_dev | <span style="color:red">76.37</span> | 74.57 |
| MME(cognition) | <span style="color:red">432</span> | 384 |
| MME(perception) | <span style="color:red">1598</span> | 1512 |
| SeedBench | <span style="color:red">68.20</span> | 66.80 |
| SeedBench_img | <span style="color:red">73.75</span> | 72.72 |
| MMStar | <span style="color:red">50.98</span> | 48.98 |
| MMMU | <span style="color:red">44.30</span> | 44.20 |
| OCRBench | <span style="color:red">531.00</span> | 525.00 |
| ChartQA | <span style="color:red">67.84</span> | 66.52 |
| DocVQA_val | <span style="color:red">76.46</span> | 75.21 |
| POPE | 88.69 | <span style="color:red">88.83</span> |
| TextVQA_val | 61.69 | <span style="color:red">62.47</span> |
B. Linear Probe Evaluation Results
This table presents the results of linear probe evaluations comparing CLIP and MLCD models on the ViT_L_14_336px architecture across various datasets. The linear probe test freezes the pre-trained model's weights and trains a linear classifier on top to assess how well the model's representations generalize to different tasks.
| Dataset | MLCD (ViT_L_14_336px) | CLIP (ViT_L_14_336px) |
|---|---|---|
| AVG | <span style="color:red">87.15</span> | 85.35 |
| Food101 | <span style="color:red">96.21</span> | 95.90 |
| CIFAR-10 | <span style="color:red">99.36</span> | 97.90 |
| CIFAR-100 | <span style="color:red">93.69</span> | 87.40 |
| Birdsnap | <span style="color:red">88.18</span> | 79.90 |
| SUN397 | <span style="color:red">87.96</span> | 82.20 |
| Stanford Cars | <span style="color:red">95.16</span> | 91.50 |
| FGVC Aircraft | <span style="color:red">86.38</span> | 71.60 |
| Describable Textures Dataset | <span style="color:red">86.70</span> | 83.00 |
| Oxford-IIIT Pets | <span style="color:red">96.27</span> | 95.10 |
| Caltech-101 | <span style="color:red">97.92</span> | 96.00 |
| Flowers102 | <span style="color:red">99.58</span> | 99.20 |
| MNIST | 98.67 | <span style="color:red">99.20</span> |
| STL-10 | 99.28 | <span style="color:red">99.70</span> |
| EuroSAT | <span style="color:red">99.06</span> | 98.10 |
| RESISC45 | <span style="color:red">95.48</span> | 94.90 |
| GTSRB | 92.32 | <span style="color:red">92.40</span> |
| KITTI | <span style="color:red">75.39</span> | 69.20 |
| Country211 | 38.12 | <span style="color:red">46.40</span> |
| PatchCamelyon | <span style="color:red">88.00</span> | 85.60 |
| UCF101 | <span style="color:red">92.86</span> | 92.00 |
| Kinetics-700 | <span style="color:red">73.35</span> | 73.00 |
| CLEVR | <span style="color:red">64.40</span> | 60.30 |
| Hateful Memes | 72.00 | <span style="color:red">77.30</span> |
| SST-2 | 76.33 | <span style="color:red">80.50</span> |
| ImageNet | <span style="color:red">86.30</span> | 85.40 |
C. Limitations
Models with higher resolution are more friendly to OCR results. We are currently training such models and will soon make them available.
Acknowledgments
We would like to express our gratitude to Xie Yin and Yumeng Wang for their significant contributions to the experimental validation in MLLMs.
- Task
- Feature Extraction
- Type
- clip_vision_model
- License
- apache-2.0
- Created
- Oct 11, 2024
- Updated
- Nov 13, 2024