autocatalogai-clip-multitask-v2

AutoCatalogAI V2

AutoCatalogAI V2 is a multi-task computer vision model for automatically extracting fashion product attributes and generating structured catalog metadata from a single product image.

The model predicts seven attributes:

• Gender
• Master category
• Subcategory
• Article type
• Base colour
• Season
• Usage

It is built on top of openai/clip-vit-base-patch32 and improves the original AutoCatalogAI V1 model through a lightweight colour-feature branch, hierarchical residual connections, controlled fine-tuning, and optional consistency correction.

Model Overview

AutoCatalogAI V2 uses a shared CLIP image encoder followed by multiple task-specific classification heads.

Product Image
      │
      ├── CLIP Vision Encoder
      │         │
      │         ├── Gender Head
      │         ├── Master Category Head
      │         ├── Subcategory Head
      │         ├── Article Type Head
      │         ├── Season Head
      │         └── Usage Head
      │
      └── Colour Statistics Branch
                │
                └── Base Colour Head

The model also uses lightweight hierarchical residual connections:

Master Category → Subcategory
Subcategory → Article Type
Article Type → Season
Article Type → Usage

These connections improve consistency between related product attributes without replacing the independent task heads.

Improvements Over V1

Compared with AutoCatalogAI V1, V2 introduces:

• A dedicated colour-feature branch
• Hierarchical residual connections between related tasks
• Two-stage fine-tuning
• Mild capped class balancing for selected tasks
• Validation-based safe checkpoint selection
• Optional consistency correction
• Separate single-image and batch latency benchmarks
• Fixed classification-report generation for classes absent from the test split

The V2 training process starts from the proven V1 checkpoint rather than training the complete model from scratch.

Dataset

The model was trained and evaluated using:

ashraq/fashion-product-images-small

Dataset split:

The final test set contains 6,611 product images.

Predicted Attributes

Corrected Test Results

The following results include lightweight consistency correction based on attribute relationships learned from the training split.

Overall Corrected Metrics

Exact-match accuracy requires all seven attributes to be predicted correctly for the same image.

Raw Model Results

The raw results below are calculated before consistency correction.

The small difference between raw and corrected performance shows that most of the improvement comes directly from the trained model rather than rule-based post-processing.

V1 and V2 Comparison

V2 improves exact-match accuracy by approximately 12.69 percentage points and base-colour accuracy by approximately 9.61 percentage points over V1.

Training Strategy

Training was performed in two stages.

Stage 1

• The CLIP image encoder remained frozen
• Newly introduced colour and hierarchy components were trained
• Selected classification heads were fine-tuned
• A stable V1-equivalent checkpoint was preserved as a fallback

Stage 2

• The final CLIP vision layer was unfrozen
• All task heads were trained using controlled learning rates
• Mild capped class weights were used only for selected imbalanced tasks
• The best checkpoint was selected using validation performance and safety constraints

The final checkpoint was selected from:

stage_2_epoch_3

Inference Performance

Latency was measured on an NVIDIA Tesla P100 GPU.

Single-Image Inference

Batch Inference

Latency depends on hardware, batch size, PyTorch version, and runtime configuration.

Repository Files

The repository contains the artifacts required to load and use the custom model:

model.pt
config.json
label_maps.json
consistency_rules.json
metrics.json
README.md

Additional evaluation reports and prediction files may also be included.

Downloading the Model

The complete repository can be downloaded using huggingface_hub:

from huggingface_hub import snapshot_download

model_directory = snapshot_download(
    repo_id="mohsin416/autocatalogai-clip-multitask-v2"
)

print(model_directory)

Loading

AutoCatalogAI V2 uses a custom PyTorch architecture and cannot be loaded directly with:

AutoModelForImageClassification.from_pretrained(...)

Use the AutoCatalogAI V2 project architecture to:

1. Load config.json
2. Load label_maps.json
3. Initialize the custom multi-task model
4. Load the state dictionary from model.pt
5. Generate colour features during preprocessing
6. Optionally apply consistency_rules.json

Example project-level usage:

from autocatalog.inference.predictor import AutoCatalogPredictor

predictor = AutoCatalogPredictor(
    repo_id="mohsin416/autocatalogai-clip-multitask-v2"
)

result = predictor.predict("product_image.jpg")

print(result["prediction"])
print(result["catalog_output"])

The predictor implementation must support the V2 colour branch and hierarchical residual layers.

Example Output

{
  "prediction": {
    "gender": {
      "label": "Women",
      "confidence": 0.8828
    },
    "masterCategory": {
      "label": "Footwear",
      "confidence": 0.9999
    },
    "subCategory": {
      "label": "Flip Flops",
      "confidence": 0.9888
    },
    "articleType": {
      "label": "Flip Flops",
      "confidence": 0.9903
    },
    "baseColour": {
      "label": "Red",
      "confidence": 0.7010
    },
    "season": {
      "label": "Summer",
      "confidence": 0.7783
    },
    "usage": {
      "label": "Casual",
      "confidence": 0.9980
    }
  },
  "catalog_output": {
    "suggested_title": "Women Red Casual Flip Flops",
    "search_tags": [
      "women flip flops",
      "red flip flops",
      "casual flip flops",
      "summer footwear",
      "women casual wear",
      "red casual fashion",
      "flip flops"
    ]
  }
}

Intended Uses

AutoCatalogAI V2 is intended for:

• Fashion product catalog automation
• Product attribute extraction
• E-commerce metadata generation
• Product title suggestion
• Search-tag generation
• Fashion dataset analysis
• Multi-task learning demonstrations
• Computer vision portfolio and research projects

Out-of-Scope Uses

The model is not intended for:

• Identifying or classifying people
• Inferring a person's gender from an image
• Safety-critical decision-making
• Legal, medical, or financial applications
• General-purpose object recognition outside the supported fashion domain
• Images containing multiple unrelated products without preprocessing

The gender output represents the dataset's product-target category and must not be interpreted as a prediction about a person.

Limitations

Base Colour

Base-colour prediction remains the most difficult task because:

• Many products contain multiple colours
• Similar labels such as red, pink, peach, and orange overlap visually
• Background and lighting affect colour statistics
• Dataset colour annotations may be subjective or noisy

The model achieves 69.72% Top-1 and 90.70% Top-3 accuracy for base colour.

Class Imbalance

The gap between average accuracy and macro F1 indicates weaker performance on rare labels. Common classes generally receive stronger predictions than underrepresented classes.

Season and Usage

Season and usage are not always directly visible in an image. These attributes can depend on product descriptions, regional conventions, and seller metadata.

Domain Shift

The model was evaluated on images from the same dataset distribution used during development. Performance may decrease on:

• Real-world marketplace photos
• Complex backgrounds
• Low-resolution images
• Occluded products
• Multiple-product images
• Unusual camera angles
• Strong lighting or colour filters

Evaluation Notes

• Label mappings were kept consistent across training, validation, and test splits
• Test metrics were calculated on 6,611 held-out images
• Top-3 accuracy checks whether the correct class appears among the three highest-probability predictions
• Exact-match accuracy requires all seven attributes to be correct
• Consistency correction uses mappings derived only from the training split
• Corrected and raw metrics are reported separately for transparency

Upstream Models and Data

AutoCatalogAI V2 depends on:

• openai/clip-vit-base-patch32
• ashraq/fashion-product-images-small

Version

Model: AutoCatalogAI V2
Architecture: CLIP ViT-B/32 multi-task classifier
Checkpoint source: AutoCatalogAI V1
Test samples: 6,611
Primary framework: PyTorch + Transformers