distilbert-base-uncased-sst2-sentiment-kd

DistilBERT Base Uncased SST-2 Sentiment Model (Distilled from BERT)

This model is a DistilBERT-base-uncased model fine-tuned on the SST-2 (Stanford Sentiment Treebank) dataset using knowledge distillation from a BERT-base teacher (textattack/bert-base-uncased-SST-2).

It achieves 90.83% validation accuracy — just 1.61% below the teacher — while being 1.6× smaller and nearly 2× faster at inference time.

Model Details

• Student architecture: distilbert-base-uncased
• Teacher model: textattack/bert-base-uncased-SST-2
• Distillation method: Logits-based knowledge distillation (Hinton et al., 2015)
• Temperature: 4.0
• Loss weighting: α = 0.7 (70% distillation loss, 30% hard-label loss)
• Training steps: 3 epochs, AdamW (lr=5e-5), batch size=16
• Reproducibility: Fixed random seed (42)

Intended Use

This model is intended for binary sentiment classification of English sentences (e.g., movie reviews). It outputs two classes:

• 0: Negative sentiment
• 1: Positive sentiment

Out-of-Scope Use

• Not suitable for non-English text
• Not designed for fine-grained sentiment (e.g., 5-star ratings)
• Not evaluated on social media text with heavy slang or emojis

Performance

• Distillation efficiency: 95.77%
• Size reduction: 38.8% fewer parameters
• Inference speedup: 1.98× faster than teacher

Usage

With Transformers Pipeline

from transformers import pipeline

classifier = pipeline(
    "sentiment-analysis",
    model="JUSTIAM/distilbert-base-uncased-sst2-sentiment-kd",
    tokenizer="JUSTIAM/distilbert-base-uncased-sst2-sentiment-kd"
)

result = classifier("This movie is absolutely fantastic!")
print(result)
# Output: [{'label': 'POSITIVE', 'score': 0.998}]

Manual Inference

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_name = "JUSTIAM/distilbert-base-uncased-sst2-sentiment-kd"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

text = "I really hate this boring film."
inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True, max_length=128)

with torch.no_grad():
    logits = model(**inputs).logits
    probabilities = torch.softmax(logits, dim=-1)
    predicted_class = torch.argmax(probabilities, dim=-1).item()

sentiment = "POSITIVE" if predicted_class == 1 else "NEGATIVE"
confidence = probabilities[0][predicted_class].item()

print(f"Sentiment: {sentiment}, Confidence: {confidence:.3f}")

Training Data

• Dataset: GLUE/SST-2
• Task: Binary sentiment classification
• Train samples: 67,349
• Validation samples: 872

Evaluation

Evaluated on the official SST-2 validation set. No test set used.