bert-base-uncased-feature-extraction-glue-mnli

BERT Feature-based Classifier for MNLI

This model is a feature-based transfer learning classifier built on top of a frozen BERT-base-uncased encoder, fine-tuned on the MNLI (Multi-Genre Natural Language Inference) dataset. Instead of fine-tuning the entire BERT model, the encoder was frozen and a custom classification head was trained on top using pooled representations of sentence pairs.

---

Model Description

• Base model: bert-base-uncased
• Approach: Feature-based transfer learning
• Task: Natural Language Inference (NLI) on MNLI dataset
• Head architecture:
  • Concatenation of [CLS] embedding and mean-pooled hidden states
  • Fully connected feed-forward classifier with LayerNorm, ReLU, Dropout layers

---

Training Details

• Dataset: GLUE: MultiNLI
• Training split size: ~392k sentence pairs
• Validation splits: validation_matched, validation_mismatched
• Frozen parameters: All BERT parameters (only classification head trained)
• Optimizer: AdamW (lr=3e-5)
• Scheduler: StepLR (decay every 10 epochs, γ=0.5)
• Loss function: CrossEntropyLoss
• Batch size: 64
• Epochs: 50
• Device: CUDA (if available)

---

Results

After 50 epochs, the model achieved the following performance:

Final Training Loss: 0.1935

Validation (Matched split)

• Accuracy: 0.7878
• F1 Score: 0.7879
• Precision: 0.7882
• Recall: 0.7878

Validation (Mismatched split)

• Accuracy: 0.7941
• F1 Score: 0.7942
• Precision: 0.7944
• Recall: 0.7941