Downloads · 30 days
0
clip-lora-fashion-search
sidmhuggingface/clip-lora-fashion-search
clip-lora-fashion-search is a zero-shot classification model from sidmhuggingface. Use it when you need labels you did not train the model on. It is set up for peft. The card lists the license as mit.
A LoRA fine-tune of openai/clip-vit-base-patch32's vision encoder, trained to improve text-to-image retrieval on the Fashion Product Images (Small) catalog (44,412 products). Built as the backbone of a multimodal sema…
Access
Public
Updated Jul 19, 2026
Repo size
488 MB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors306 MB · 62%
At a glance
- Task
- Zero-Shot Classification
- Library
- peft
- License
- mit
- Access
- Public
- Created
- Jul 19, 2026
- Updated
- Jul 19, 2026
- SHA
- fc1dc4ab
Try a prompt
Base models
Model card
From the Hugging Face model README
CLIP ViT-B/32 LoRA fine-tuned on Fashion Product Images
A LoRA fine-tune of openai/clip-vit-base-patch32's vision encoder, trained to improve
text-to-image retrieval on the Fashion Product Images
(Small) catalog
(44,412 products). Built as the backbone of a multimodal semantic search + zero-shot tagging
project. Full code, tests, and write-up: GitHub
repo.
What's in this repo
| Path | Contents |
|---|---|
clip_merged_fp16/ | The recommended way to use this model: base CLIP + LoRA merged into one checkpoint, saved in native fp16. Load directly with CLIPModel.from_pretrained(...), no peft needed. |
lora_adapter/ | The LoRA adapter alone (rank 16, vision encoder q_proj/k_proj/v_proj only, text encoder untouched), for use with peft.PeftModel.from_pretrained on top of the base fp32 checkpoint. |
embeddings_pretrained.npy / embeddings_lora.npy | Precomputed CLIP embeddings for all 44,412 catalog images, before and after fine-tuning, aligned to item_ids.npy. |
styles_clean.csv | Cleaned product metadata (id, product name, article type, color) matching the embeddings. |
eval_results.json | Recall@1/5/10 and MRR, pretrained vs. fine-tuned, on 40 held-out color+category query phrases never seen during training. |
split.json, loss_history.json | Train/eval id split and per-step training loss, for reproducibility. |
How to use (recommended: merged fp16 checkpoint)
from transformers import CLIPModel, CLIPProcessor
model = CLIPModel.from_pretrained("sidmhuggingface/clip-lora-fashion-search", subfolder="clip_merged_fp16")
processor = CLIPProcessor.from_pretrained("sidmhuggingface/clip-lora-fashion-search", subfolder="clip_merged_fp16")
Training details
- Base model:
openai/clip-vit-base-patch32 - Method: LoRA (rank 16, alpha 32, dropout 0.1) applied only to the vision encoder's
q_proj/k_proj/v_projattention projections. Text encoder frozen entirely. - Objective: CLIP contrastive loss (InfoNCE, in-batch negatives) on (image,
productDisplayName) pairs. - Data: 44,412 product images, 90% train / 10% id-level holdout (seed 42).
- Hardware: ASU Sol HPC, 1x NVIDIA A100-SXM4-80GB, 2 epochs, well under 10 minutes of GPU compute.
Evaluation
Measured on 40 held-out (color, article type) query phrases (e.g. "black formal shoes") never seen during fine-tuning, each checked against the full 44,412-item catalog using a from-scratch Recall@k / MRR implementation.
| Model | Recall@1 | Recall@5 | Recall@10 | MRR |
|---|---|---|---|---|
| Pretrained CLIP | 1.4% | 7.6% | 14.0% | 0.704 |
| LoRA fine-tuned | 1.6% | 9.1% | 17.2% | 0.745 |
Recall@10 improved +23.0% relative to the pretrained baseline. Absolute Recall@10 looks modest because these queries have large relevant sets (25-107 relevant images per query out of 44,412) — MRR (whether the first correct hit ranks early) is the more forgiving and arguably more representative metric for this catalog's query style, and improved +5.8% relative.
Limitations
CLIP has known weaknesses on compositional attribute binding — e.g. a query like "red dress" can return visually similar red garments (nightdresses, camisoles) rather than dresses specifically, consistent with findings in Thrush et al., 2022 (Winoground). This fine-tune improves retrieval quality on this catalog's query distribution but does not fix that underlying limitation.
License
MIT, matching the base openai/clip-vit-base-patch32 model.
- Task
- Zero-Shot Classification
- Library
- peft
- License
- mit
- Created
- Jul 19, 2026
- Updated
- Jul 19, 2026