hw2-text-distilbert

DistilBERT Text Classification Model

This model is a fine-tuned version of distilbert-base-uncased for text classification tasks.

Model Description

This model is a fine-tuned DistilBERT model for binary text classification, specifically designed to classify text as being related to either Pittsburgh or Shanghai cities. The model achieves excellent performance with 99.5% accuracy on the test set.

• Model type: Text Classification (Binary)
• Language(s) (NLP): English
• Base model: distilbert-base-uncased
• Classes: Pittsburgh, Shanghai

Intended Uses & Limitations

Intended Uses

• Binary text classification between Pittsburgh and Shanghai-related content
• City-based text categorization tasks
• Research and educational purposes in NLP and text classification

Limitations

• Limited to English language text
• Performance may vary on out-of-domain data
• Maximum input length of 256 tokens due to truncation

Training and Evaluation Data

Training Data

• Base dataset: cassieli226/cities-text-dataset
• Classes: Pittsburgh (507 samples) and Shanghai (493 samples) in augmented dataset
• Original dataset: 100 samples (50 Pittsburgh, 50 Shanghai)
• Data augmentation: Applied to increase dataset size from 100 to 1000 samples
• Train/Test Split: 80/20 split (800 train, 200 test) with stratified sampling
• External validation: Original 100 samples used for additional validation

Preprocessing

• Text tokenization using DistilBERT tokenizer
• Maximum sequence length: 256 tokens
• Truncation applied to longer sequences

Training Procedure

Training Hyperparameters

• Learning rate: 5e-5
• Training batch size: 16
• Evaluation batch size: 32
• Number of epochs: 4
• Weight decay: 0.01
• Warmup ratio: 0.1
• LR scheduler: Linear
• Gradient accumulation steps: 1
• Mixed precision: FP16 (if GPU available)

Training Configuration

• Optimizer: AdamW (default)
• Early stopping: Enabled with patience of 2 epochs
• Best model selection: Based on F1 score (macro)
• Evaluation strategy: Every epoch
• Save strategy: Every epoch (best model only)

Evaluation

Metrics

The model was evaluated using:

• Accuracy: Overall classification accuracy
• F1 Score (Macro): Macro-averaged F1 score across all classes
• Per-class accuracy: Individual class performance metrics

Results

• Test Set Performance:
  • Accuracy: 99.5%
  • F1 Score (Macro): 99.5%
• External Validation:
  • Accuracy: 100.0%
  • F1 Score (Macro): 100.0%

Detailed Performance

• Pittsburgh Class: 99.01% accuracy (101 samples)
• Shanghai Class: 100.0% accuracy (99 samples)
• Confusion Matrix: Only 1 misclassification out of 200 test samples

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

# Load model and tokenizer
model_name = "Anyuhhh/hw2-text-distilbert"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

# Example usage
text = "Your input text here"
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=256)

with torch.no_grad():
    outputs = model(**inputs)
    predictions = torch.nn.functional.softmax(outputs.logits, dim=-1)
    predicted_class = torch.argmax(predictions, dim=-1)

print(f"Predicted class: {predicted_class.item()}")