SINC-V1-SIGLIP2-KEE-SPEED-small

SINC-V1-SIGLIP2-KEE-SPEED-Small: Multimodal Product Classifier

Shopify Image Niche Classifier (V1-small) : A high-performance multimodal product classification model that combines text (product titles) and images to classify products into 14 different categories.

Model Performance

• Test Accuracy: 92.46%
• Test Macro F1-Score: 0.73
• Test Loss: 0.2450
• Test Set Size: 59,789 samples

Per-Class Performance

Model Architecture

• Text Encoder: microsoft/deberta-v3-small
• Image Encoder: google/siglip2-base-patch16-256
• Fusion: Concatenation + MLP (512 hidden units)
• Output: 14-class classifier

Usage

from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer, AutoProcessor, SiglipModel
import torch
from PIL import Image
import json
from model_arch import MultimodalClassifier

# Download model and config from Hugging Face
repo_id = "manavbangotra/sinc-v1-siglip2-kee-speed-small"  # Replace with your repo
model_path = hf_hub_download(repo_id=repo_id, filename="best_multimodal.pt")
config_path = hf_hub_download(repo_id=repo_id, filename="config.json")

# Load configuration
with open(config_path) as f:
    config = json.load(f)

labels = config["labels"]
num_labels = config["num_labels"]

# Load model components
tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-v3-small")
processor = AutoProcessor.from_pretrained("google/siglip2-base-patch16-256")
clip_model = SiglipModel.from_pretrained("google/siglip2-base-patch16-256")

model = MultimodalClassifier("microsoft/deberta-v3-small", clip_model, num_labels, text_finetune=True, clip_finetune=False)
#Use cuda if available
model.load_state_dict(torch.load(model_path, map_location="cpu"))

# Example usage
text = "Elegant summer dress with floral pattern"
image = Image.open("product_image.jpg")

text_inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True, max_length=64)
image_inputs = processor(images=image, return_tensors="pt")

# Predict
with torch.no_grad():
    outputs = model(
        input_ids=text_inputs["input_ids"],
        attention_mask=text_inputs["attention_mask"],
        pixel_values=image_inputs["pixel_values"]
    )
    predictions = torch.softmax(outputs["logits"], dim=-1)
    predicted_class = torch.argmax(predictions, dim=-1).item()
    confidence = predictions[0][predicted_class].item()

print(f"Predicted: {labels[predicted_class]} ({confidence:.2%})")

Categories

The model classifies products into these 14 categories:

1. Baby & Kids
2. Beauty & Personal Care
3. Consumer Electronics
4. Fashion
5. Food & Beverages
6. Health & Supplements
7. Hobbies & Collectibles
8. Home & Interior
9. Jewelry
10. Lights
11. Office Supplies
12. Others
13. Outdoor, Garden & Adventure Gear
14. Sports & Fitness Equipment

Training Details

• Training Data: E-commerce product dataset with titles and images
• Training Strategy: Fine-tuned text encoder, frozen image encoder
• Optimizer: AdamW with linear warmup scheduler
• Batch Size: 8
• Learning Rate: 2e-5
• Epochs: 3
• Max Text Length: 64 tokens

Limitations

• Performance varies across categories (especially small classes)
• Requires mandatory image input.
• Trained on specific Shopify product domain

Citation

@misc{sinc-v1-siglip2-kee-speed-small,
  title={SINC-V1-SIGLIP2-KEE-SPEED-small: High-Performance Multimodal Product Classifier},
  author={Manav Bangotra},
  year={2025},
  url={https://huggingface.co/manavbangotra/sinc-v1-siglip2-kee-speed-small}
}

License

MIT License