distilbert-imdb-sentiment-classifier-lora

DistilBERT IMDb Sentiment Classifier (LoRA Fine-Tuned)

This project demonstrates parameter-efficient fine-tuning of DistilBERT using LoRA (PEFT) for sentiment classification on the IMDb dataset.

---

Model Details

• Base Model: distilbert-base-uncased
• Task: Binary Sentiment Classification (Positive / Negative)
• Fine-tuning Method: LoRA (Low-Rank Adaptation)
• Final Model: LoRA merged into base model (standalone)

---

Training Setup

• Learning Rate: 1.5e-5
• Scheduler: Cosine
• Warmup Ratio: 0.1
• Epochs: 4
• Batch Size: 8 (grad accumulation = 4)
• Weight Decay: 0.01
• FP16: Enabled

---

LoRA Configuration

• Target Modules: q_lin, k_lin, v_lin
• Rank (r): 16
• Alpha: 32
• Dropout: 0.05

---

Evaluation Metrics

• Accuracy: ~83%
• Precision: ~0.79
• Recall: ~0.91
• F1 Score: ~0.85

---

Key Observations

• LoRA significantly reduces training cost while maintaining strong performance
• Expanding LoRA to Q, K, V improves attention adaptation
• Lower learning rate improves stability and generalization

---

Known Limitations

• Slight bias toward positive class (high recall)

Challenges faced and fixes implemented

---

Challenges & Fixes

• Limited adaptation due to low-rank constraint
  With a smaller rank (r=8), the model wasn’t learning meaningful patterns — updates were too constrained.
  Fix: Increased LoRA rank → r=16, trained on full IMDB dataset (25k samples), used max_length=256
  Why it worked: Higher rank increased LoRA capacity → allowed richer updates instead of minimal adjustments

• Learning rate sensitivity (instability vs slow learning)
  Higher LR like 2e-4 led to unstable/noisy training. Too low LR made learning too slow.
  Fix: Settled on learning_rate = 1.5e-5, added warmup_steps=200, used weight_decay=0.01
  Why it worked: Ensured stable, incremental updates within LoRA’s constrained update space

• Balanced hyperparameter tuning (r, alpha, LR)
  Changing one parameter didn’t help unless others were balanced.
  Fix: r=16, lora_alpha=16, lora_dropout=0.1
  Why it worked: Maintained stable scaling of updates (alpha / r) while increasing capacity

• Going beyond standard evaluation metrics
  Accuracy/F1 looked fine, but the model leaned slightly toward positive predictions.
  Fix: Checked confusion matrix, tuned decision threshold from 0.5 → 0.6
  Why it worked: Reduced false positives → improved prediction balance and calibration

• Ambiguous sample data labelling
  Sentences like “not bad” or “okay” were labelled negative but model learned positive
  Fix: Manual error analysis, treated them as dataset ambiguity instead of forcing the model to fit them
  Why it worked: Avoided overfitting to noisy or subjective labels

• Verifying real improvement beyond metrics
  Metric gains didn’t guarantee better behavior. On paper the metrics looked good but evaluation had to go beyond the metrics.
  Fix: Tested on long & mixed sentiment inputs, compared outputs with pretrained baseline, reviewed actual predictions
  Why it worked: Confirmed improvements were behavioral (generalization), not just numerical

  ---

Results & Comparison

Observations

• LoRA fine-tuned model achieved strong performance on IMDb:

  • Accuracy: 88.47%
  • Balanced precision (0.877) and recall (0.895)

• Compared to pretrained behavior:

  • Shows lower overconfidence on predictions
  • Handles long and mixed reviews more realistically

• Cross-domain evaluation (SST-2):

  • Accuracy: 83.83%
  • Expected drop due to domain shift
  • Recall remains high (0.91), indicating sensitivity to positive sentiment

• Model behavior insights:

  • More calibrated on ambiguous inputs (scores closer to decision boundary)
  • Some misclassification in long reviews due to presence of mixed sentiment

Usage

from transformers import AutoModelForSequenceClassification, AutoTokenizer

model = AutoModelForSequenceClassification.from_pretrained("nidhic0301/distilbert-imdb-sentiment-classifier-lora")
tokenizer = AutoTokenizer.from_pretrained("nidhic0301/distilbert-imdb-sentiment-classifier-lora")