Downloads · 30 days
12
28% of all-time downloads
clip-vit-base-patch32-coco-retrieval
aaronwzl/clip-vit-base-patch32-coco-retrieval
clip-vit-base-patch32-coco-retrieval is a zero-shot image classification model from aaronwzl. Use it for the zero-shot image classification task on the model card, and read the license before you ship it in a product. The card lists the license as mit.
The retrieval evaluation uses specific stream indices of sayakpaul/coco-30-val-2014. The training script streams past those and takes only samples whose index is not in the evaluation set, and asserts the disjointness…
All-time downloads
43
Public
Parameters
151M
605 MB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors605 MB · 99%
At a glance
- Task
- Zero-Shot Image Classification
- License
- mit
- Model type
- clip
- Access
- Public
- Created
- Aug 29, 2026
- Updated
- Aug 29, 2026
- SHA
- 750a1ccc
Base models
Model card
From the Hugging Face model README
Training
| base model | openai/clip-vit-base-patch32 |
| objective | symmetric InfoNCE over (image, caption) pairs, using CLIP's learned temperature — the pre-training objective continued at a low learning rate |
| data source | sayakpaul/coco-30-val-2014 (streaming split train) |
| training pairs | 4000 |
| epochs | 2 |
| optimiser steps | 126 |
| batch size | 64 |
| optimiser | AdamW, lr 1e-05, weight decay 0.1 |
| schedule | 10 % linear warmup, then linear decay to 0 |
| gradient clipping | global norm 1.0 |
| seed | 42 |
| device | cuda |
| wall-clock | 345 s |
| loss | 0.4031 → 0.0519 |
Evaluation-set leakage control
The retrieval evaluation uses specific stream indices of sayakpaul/coco-30-val-2014. The training script
streams past those and takes only samples whose index is not in the evaluation set,
and asserts the disjointness at run time rather than assuming it.
Train/eval index overlap = 0 (verified, 500 evaluation indices held out against 4000 training indices).
Both index lists are shipped in finetune_config.json, so the disjointness is checkable
without rerunning anything.
Effect of fine-tuning
Mean cosine similarity on a held-out sample of evaluation pairs, ground-truth versus foil caption:
| zero-shot | fine-tuned | margin change | |
|---|---|---|---|
| MSCOCO | GT 0.3245 / foil 0.1490 | GT 0.3165 / foil 0.0976 | +0.0434 |
Fine-tuning widens the ground-truth-versus-foil margin mainly by pushing foil similarity down.
Usage
from transformers import CLIPModel, CLIPProcessor
model = CLIPModel.from_pretrained("aaronwzl/clip-vit-base-patch32-coco-retrieval")
proc = CLIPProcessor.from_pretrained("aaronwzl/clip-vit-base-patch32-coco-retrieval")
Reproduction
python rebuttal/exp/retrieval/finetune_clip.py \
--dataset coco --n-train 4000 --epochs 2 \
--batch-size 64 --lr 1e-05 --seed 42
- Task
- Zero-Shot Image Classification
- Type
- clip
- License
- mit
- Created
- Aug 29, 2026
- Updated
- Aug 29, 2026