bert-base-uncased-LoRA-quora-question-pairs

BERT Base Uncased + LoRA Fine-Tuned For Quora Duplicate Question Detection

This model applies LoRA (Low-Rank Adaptation) fine-tuning on tomaarsen/bert-base-nq-prompts for the Quora Question Pairs dataset.

It classifies whether two questions are duplicates.

---

Model Details

• Base Model: tomaarsen/bert-base-nq-prompts
• Fine-tuning Method: LoRA (PEFT)
• LoRA Config:
  • r=8, alpha=64, dropout=0.1
  • Target modules: query, key, value, dense
• Dataset: Quora Question Pairs
• Training Epochs: 16
• Optimizer: AdamW (torch fused)
• Batch Size: 64 (gradient accumulation = 2)
• Loss Function: CrossEntropyLoss

---

Performance (Epoch 16)

---

Example Usage

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
from peft import PeftModel

# Load base model and tokenizer
model_name = "tomaarsen/bert-base-nq-prompts"
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Load LoRA fine-tuned model
base_model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=2)
model = PeftModel.from_pretrained(base_model, "mia-project-2025/bert-base-uncased-LoRA-quora-question-pairs")

model.eval()

def predict_duplicate(q1, q2):
    inputs = tokenizer(q1, q2, return_tensors="pt", truncation=True, padding=True, max_length=128)
    with torch.no_grad():
        logits = model(**inputs).logits
        pred = torch.argmax(logits, dim=1).item()
    return "Duplicate" if pred == 1 else "Not Duplicate"

# Example
print(predict_duplicate("How can I learn Python?", "What are the best ways to learn Python programming?"))