Junyi-BERT-sentiment-analysis-SST5

Source Code

• https://github.com/shjy2008/BERT

Task

• 5-class sentiment classification (labels: 0–4)

Base Model

• google-bert/bert-base-uncased (~110M parameters)

Training Data

• Stage 1: Large-scale sentiment pre-finetuning

  • Dataset: 3M-sample sentiment corpus composed of:
    • Amazon Reviews 2023 (Movies and TV): 2.34M samples
      • https://amazon-reviews-2023.github.io/
    • Rotten Tomatoes movies and critic reviews: 634k samples
      • https://www.kaggle.com/datasets/stefanoleone992/rotten-tomatoes-movies-and-critic-reviews-dataset
    • SST-5 training set: 8k samples
      • https://huggingface.co/datasets/SetFit/sst5
  • Task: 5-class sentiment classification (labels: 0–4)

• Stage 2: SST-5 sentiment fine-tuning

  • Dataset: SST-5 training set (8k samples)
    • https://huggingface.co/datasets/SetFit/sst5
  • Task: 5-class sentiment classification (labels: 0–4)

Training Method

• Loss function: CORAL (COnsistent RAnk Logits)
  • https://www.sciencedirect.com/science/article/pii/S016786552030413X
• Weight decay: 0.01
• Batch size: 32
• Dropout: 0.2
• Learning rate: 5e-5

Evaluation

• Test dataset: SST-5 test set (~2k samples)
  • https://huggingface.co/datasets/SetFit/sst5
• Test accuracy: 59.4%

Comparison

• Despite its smaller size, the model achieves competitive performance compared to significantly larger models:
  • Heinsen Routing + RoBERTa-large (~355M parameters): 59.1%
  • BERT-large (~340M parameters): 55.5%