fashion-product-classifier-vit

Fashion Product Classifier — Fine-Tuned ViT

Model

This repository contains a custom multi-task Vision Transformer (ViT) fine-tuned for fashion product attribute classification.

The pretrained base model used is:

google/vit-base-patch16-224

The original pretrained ViT provides the shared visual representation. Its weights were then fine-tuned on the cleaned Fashion Product Images (Small) dataset to adapt the model specifically for fashion product labeling.

---

Architecture

The model uses a shared ViT backbone with five task-specific classification heads.

Input Image
   │
   │ 224 × 224 RGB
   ▼
┌──────────────────────────────┐
│ google/vit-base-patch16-224  │
│     Shared ViT Backbone      │
└──────────────┬───────────────┘
               │
               │ 768-dimensional features
               ▼
      ┌────────┼────────┬────────────┬───────────┐
      ▼        ▼        ▼            ▼           ▼
   Master   SubCategory ArticleType BaseColour  Usage
   Head        Head        Head        Head      Head
     │          │           │           │          │
     ▼          ▼           ▼           ▼          ▼
   4 cls      26 cls       54 cls      46 cls     7 cls

Input

• Image: RGB image
• Model input size: 224 × 224
• Preprocessing: Hugging Face ViT image processor

Output

The model produces five independent classification outputs from the same shared visual representation:

This design allows a single image to receive multiple related fashion attributes in one forward pass.

---

Fine-Tuning Strategy

The model was not trained from scratch.

Instead:

1. Start with the pretrained google/vit-base-patch16-224.
2. Use its learned visual representations as the shared backbone.
3. Add five custom classification heads for the fashion tasks.
4. Fine-tune the model on the cleaned fashion dataset.
5. Use class-weighted CrossEntropyLoss to give greater importance to underrepresented classes.

The final checkpoint therefore represents the fine-tuned custom multi-task model, not the original Google ViT checkpoint.

---

Training Environment

Training was performed in Google Colab using a Tesla T4 GPU.

Main configuration:

Optimizer               : AdamW
ViT learning rate       : 2e-5
Classification heads LR : 1e-4
Weight decay             : 0.01
Batch size               : 32
Epochs                   : 4
Mixed precision (AMP)    : Enabled
Scheduler                : CosineAnnealingLR
Early stopping           : Used

---

Model Evaluation

The final class-weighted model achieved:

Average Test Macro F1: 75.45%

Task-level Macro F1:

Master Category : 99.91%
Sub Category    : 92.21%
Article Type    : 88.37%
Base Colour     : 37.50%
Usage           : 59.28%

Macro F1 was used as the primary comparison metric because the dataset contains significant class imbalance.

---

What This Model Is For

The model is designed for fashion product image labeling, particularly for e-commerce and online catalog workflows.

Given a product image, it can automatically generate structured fashion attributes such as:

{
  "masterCategory": "...",
  "subCategory": "...",
  "articleType": "...",
  "baseColour": "...",
  "usage": "..."
}

The application layer built around this model additionally supports Top-K predictions with confidence scores.

---

Important Note

This model is a classifier, not an object detector.

It is designed primarily for one fashion product per image. Images containing multiple products or complete outfits may produce mixed or ambiguous predictions because the model generates one set of attributes for the complete image.

The associated application also uses a separate CLIP-based clothing/non-clothing guardrail before sending an image to this ViT classifier. CLIP is not part of this uploaded ViT checkpoint.

---

Model Files

The uploaded checkpoint:

class_weighted_vit_model.pth

contains the trained state of the custom multi-task ViT, including the fine-tuned backbone and task-specific classification heads.

---

Intended Use

This model is intended for:

• E-commerce product cataloging
• Automated fashion metadata generation
• Fashion image classification experiments
• Computer vision / transfer learning research
• Prototyping fashion search and recommendation systems