sign-identification-automl

HW1 Sign Identification with AutoGluon

Model Description

This repository contains an AutoML image classification model trained with AutoGluon Multimodal to identify two categories of sign images. The model was trained as part of Homework 1 in CMU 24-679 (Designing and Deploying AI/ML).

• Framework: AutoGluon Multimodal
• Backbone: TimmAutoModelForImagePrediction (~194M parameters)
• Task: Binary image classification (label)
• Classes: 0, 1

---

Training Setup

• Dataset: ecopus/sign_identification
  • augmented split (385 samples) → used for training/validation (80/20).
  • original split (35 samples) → reserved for final testing.
• Time budget: 300 seconds (≈7 minutes).
• Hardware: Colab GPU (CUDA 12.6, mixed precision).
• Presets: best_quality (ensembling + hyperparameter tuning).

---

Results

• Validation ROC-AUC: 0.998
• Test Accuracy: 97.1%
• Weighted F1: 97.1%

Classification Report (Test Set)

      precision    recall  f1-score   support
   0       0.95      1.00      0.97        19
   1       1.00      0.94      0.97        16

accuracy 0.97 35

macro avg 0.97 0.97 0.97 35 weighted avg 0.97 0.97 0.97 35

---

How to Use

Install requirements

pip install autogluon.multimodal huggingface_hub cloudpickle

import cloudpickle
from huggingface_hub import hf_hub_download

pkl_path = hf_hub_download(
    repo_id="cassieli226/sign-identification-automl",
    filename="autogluon_predictor.pkl",
    repo_type="model"
)

with open(pkl_path, "rb") as f:
    predictor = cloudpickle.load(f)

# Predict on new data (expects DataFrame with 'image' column containing file paths)
import pandas as pd
X_test = pd.DataFrame({"image": ["path/to/your/image.png"]})
print(predictor.predict(X_test))


import pathlib, shutil, zipfile
from huggingface_hub import hf_hub_download
import autogluon.multimodal as ag
import pandas as pd

zip_path = hf_hub_download(
    repo_id="cassieli226/sign-identification-automl",
    filename="autogluon_predictor_dir.zip",
    repo_type="model"
)

extract_dir = pathlib.Path("predictor_dir")
if extract_dir.exists():
    shutil.rmtree(extract_dir)
with zipfile.ZipFile(zip_path, "r") as zf:
    zf.extractall(str(extract_dir))

predictor = ag.MultiModalPredictor.load(str(extract_dir))
print(predictor.predict(pd.DataFrame({"image": ["path/to/your/image.png"]})))

---

#Intended Use
 - Coursework demonstration of AutoML for neural networks on images.
 - Educational example for using augmented vs. original splits for training and evaluation.
#Limitations
 - Trained on a small student-collected dataset (≈420 images).
 - Accuracy may not generalize to unseen real-world data.
 - Model assumes binary labels only (0, 1).
#Ethical Notes
 - Dataset is non-sensitive, contains no personal information.
 - Augmentation was applied responsibly to avoid unrealistic samples.
# References
 - Dataset: ecopus/sign_identification
 - Framework: AutoGluon
 - OpenAI’s ChatGPT (2025) was used for code generation, structuring, and debugging.