bert-base-uncased-LoRA-glue-mnli

BERT-base + LoRA on GLUE MNLI

This repository contains a BERT-base model fine-tuned with LoRA adapters on the GLUE MNLI dataset.
The model is trained for natural language inference (NLI) with three labels: entailment, neutral, and contradiction.

---

Dataset

• Name: GLUE Multi-Genre Natural Language Inference (MNLI)
• Task: Natural Language Inference
• Size: 392k training examples, with validation splits for matched (in-domain) and mismatched (out-of-domain).
• Labels:
  • 0 = Contradiction
  • 1 = Entailment
  • 2 = Neutral

---

Training Setup

• Base Model: bert-base-uncased

• Fine-tuning Method: Parameter-Efficient Fine-Tuning (PEFT) using LoRA adapters

• LoRA Configuration:

  • Rank r = 8
  • Alpha = 64
  • Target modules = query, key, value, dense
  • Dropout = 0.1

• Hyperparameters:

  • Epochs: 15
  • Batch size: 64
  • Learning rate: 5e-5
  • Optimizer: AdamW (fused)
  • Gradient accumulation: 2
  • Weight decay: 0.01
  • Warmup ratio: 0.1

• Hardware: Trained on GPUs with mixed precision (fp16).

---

Results

Final Evaluation (Epoch 15)

---

Usage

You can load this model and tokenizer using Hugging Face Transformers:

from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_name = "mia-project-2025/bert-base-uncased-LoRA-glue-mnli"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

text = {"premise": "The cat is sleeping on the sofa.", "hypothesis": "The cat is awake."}
inputs = tokenizer(text["premise"], text["hypothesis"], return_tensors="pt")
outputs = model(**inputs)
pred = outputs.logits.argmax(-1).item()
print(pred)  # 0=Contradiction, 1=Entailment, 2=Neutral