distilbert-smsafe

🛡️ Model Card for alusci/distilbert-smsafe

A lightweight DistilBERT model fine-tuned for spam detection in SMS messages. The model classifies input messages as either spam or ham (not spam), using a custom dataset of real-world OTP (One-Time Password) and spam SMS messages.

---

Model Details

Model Description

• Developed by: alusci
• Model type: Transformer-based binary classifier
• Language(s): English
• License: Apache 2.0
• Finetuned from model: distilbert-base-uncased

Model Sources

• Repository: https://huggingface.co/alusci/distilbert-smsafe

---

🛠️ Uses

Direct Use

• Detect whether an SMS message is spam or ham (OTP or not).
• Useful in prototypes, educational settings, or lightweight filtering applications.

from transformers import pipeline

classifier = pipeline("text-classification", model="alusci/distilbert-smsafe")
result = classifier("Your verification code is 123456. Please do not share it with anyone.")

# Optional: map the label to human-readable terms
label_map = {"LABEL_0": "ham", "LABEL_1": "spam"}
print(f"Label: {label_map[result[0]['label']]} - Score: {result[0]['score']:.2f}")

Out-of-Scope Use

• Not intended for email spam detection or multilingual message filtering.
• Not suitable for production environments without further testing and evaluation.

---

🧪 Bias, Risks, and Limitations

• The model may reflect dataset biases (e.g., message structure, language patterns).
• It may misclassify legitimate OTPs or non-standard spam content.
• Risk of false positives in edge cases.

Recommendations

• Evaluate on your own SMS dataset before deployment.
• Consider combining with rule-based or heuristic systems in production.

---

📚 Training Details

Training Data

• Dataset used: alusci/sms-otp-spam-dataset
• Binary labels for spam and non-spam OTP messages

Training Procedure

• Epochs: 5
• Batch Size: 16 (assumed)
• Loss Function: CrossEntropyLoss
• Optimizer: AdamW
• Tokenizer: distilbert-base-uncased

---

📈 Evaluation

Metrics

• Accuracy, Precision, Recall, F1-score on held-out validation set
• Binary classification labels:
  • LABEL_0 → ham
  • LABEL_1 → spam

Results

Evaluation metrics after 5 epochs:

• Loss: 0.2962
• Accuracy: 91.35%
• Precision: 90.26%
• Recall: 100.00%
• F1-score: 94.88%

Performance:

• Evaluation runtime: 4.37 seconds
• Samples/sec: 457.27
• Steps/sec: 9.15

---

🌱 Environmental Impact

• Hardware Type: Apple Silicon MPS GPU (Mac)
• Hours used: <1 hour (small dataset)
• Cloud Provider: None (trained locally)
• Carbon Emitted: Minimal due to local and efficient hardware

---

🔧 Technical Specifications

Model Architecture and Objective

• Base: DistilBERT
• Objective: Binary classification head on pooled output
• Parameters: ~66M (same as distilbert)

---

📬 Model Card Contact

For questions or feedback, please contact via Hugging Face profile.