bert-base-uncased-adapter-glue-mnli

BERT-base + Prefix-Tuning on GLUE MNLI

This repository provides a BERT-base model fine-tuned with Prefix-Tuning (PEFT) on the GLUE MNLI dataset for Natural Language Inference (NLI). Given a premise and a hypothesis, the model predicts one of three relations: entailment, neutral, or contradiction.

---

Dataset

• Name: GLUE MultiNLI (MNLI)
• Task: Natural Language Inference (3-way classification)
• Splits used: train, validation_matched (in-domain), validation_mismatched (out-of-domain)
• Tokenization: bert-base-uncased, pairwise (premise, hypothesis), max_length=256

Note: Label indices follow the order used by the fine-tuned head. When using the model, prefer model.config.id2label to map indices to strings.

---

Model & Method

• Base model: bert-base-uncased
• Fine-tuning method: Parameter-Efficient Fine-Tuning with Prefix-Tuning (via Hugging Face PEFT)
• Prefix-Tuning configuration:
  • num_virtual_tokens = 20
  • prefix_projection = True
  • Applies across all encoder layers/heads (using the model’s hidden size and attention heads)

---

Training Setup

• Epochs: 20
• Train batch size: 32
• Eval batch size: 16
• Learning rate: 5e-5
• Optimizer: adamw_torch_fused
• Scheduler: cosine
• Gradient accumulation: 2
• Weight decay: 0.01
• Warmup ratio: 0.1
• Mixed precision: FP16 when GPU available
• Evaluation & saving: per epoch; best model tracked by f1

---

Results

Final metrics reported after training (epoch 20):