bert-base-uncased-feature-extraction-quora-question-pairs

BERT Feature-Based Quora Duplicate Question Detection

This model uses feature-based transfer learning with tomaarsen/bert-base-nq-prompts as a frozen encoder, combined with a lightweight feedforward classifier trained on the Quora Question Pairs dataset.

It classifies whether two questions are duplicates or not.

---

Model Details

• Base Encoder: tomaarsen/bert-base-nq-prompts (frozen)
• Approach: Feature-based transfer learning (CLS embeddings extraction)
• Classifier: 2-layer feedforward neural network
• Dataset: Quora Question Pairs
• Training Epochs: 25
• Optimizer: Adam (lr=3e-5)
• Loss Function: CrossEntropyLoss

---

Performance (Epoch 25)

---

Example Usage

import torch
from transformers import AutoTokenizer
from torch import nn

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained("tomaarsen/bert-base-nq-prompts")
model = torch.load("bert_feature_based_quora.pt", map_location="cpu")
model.eval()

def predict_duplicate(q1, q2):
    # Extract embeddings
    inputs1 = tokenizer(q1, return_tensors="pt", padding=True, truncation=True)
    inputs2 = tokenizer(q2, return_tensors="pt", padding=True, truncation=True)

    with torch.no_grad():
        # Load frozen encoder
        encoder = torch.load("bert_feature_based_quora_weights.pt", map_location="cpu")
        emb1 = encoder(**inputs1).last_hidden_state[:, 0, :]
        emb2 = encoder(**inputs2).last_hidden_state[:, 0, :]
        features = torch.cat([emb1, emb2], dim=1)

        # Predict
        logits = model(features)
        pred = torch.argmax(logits, dim=1).item()

    return "Duplicate" if pred == 1 else "Not Duplicate"

# Example
print(predict_duplicate("How can I learn Python?", "What are the best ways to learn Python programming?"))