bert-tiny-semantic-similarity

bert-tiny Semantic Similarity / Paraphrase Detection

A fine-tuned prajjwal1/bert-tiny (4.4M parameters, 2 layers, hidden size 128) cross-encoder for sentence-pair semantic similarity / paraphrase / duplicate-question detection. Trained jointly on STS-B, QQP, PAWS, and MRPC.

Model Details

• Base model: prajjwal1/bert-tiny (L=2, H=128, ~4.4M params)
• Architecture: Cross-encoder, BertForSequenceClassification (2-class)
• Task: Binary classification — given two sentences, predict whether they are semantically similar / duplicate / paraphrase (label 1) or not (label 0)
• Max sequence length: 64 tokens (sentence1 + sentence2 combined)
• License: MIT (matches base model)

Training Data

Trained on the concatenation of four datasets (label conversion applied where needed):

License note: QQP's own terms are primarily intended for research/non-commercial use. If you plan commercial deployment, please review QQP's original license independently — the fine-tuned weights here are MIT-licensed, but the training-data provenance is documented for transparency.

Training Configuration

• Optimizer: AdamW, learning rate 1e-4
• Batch size: 32
• Epochs: 10
• Loss: Class-weighted CrossEntropyLoss (weights inverse-proportional to label frequency)
• Label smoothing: 0.1
• Warmup ratio: 0.1, weight decay: 0.01

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

tokenizer = AutoTokenizer.from_pretrained("YOUR_USERNAME/YOUR_MODEL_NAME")
model = AutoModelForSequenceClassification.from_pretrained("YOUR_USERNAME/YOUR_MODEL_NAME")
model.eval()

def similarity(sent1, sent2):
    inputs = tokenizer(sent1, sent2, return_tensors="pt", truncation=True, max_length=64, padding="max_length")
    with torch.no_grad():
        logits = model(**inputs).logits
        prob = torch.softmax(logits, dim=-1)[0][1].item()
    return prob  # probability that the two sentences are similar/duplicate

print(similarity("How old are you?", "What is your age?"))
# -> high probability (similar)

Evaluation Results

Evaluated on the official validation/test splits of each dataset (threshold = 0.5):

Known Limitation — PAWS / Adversarial Word-Order Sensitivity

This model, like most compact (sub-15M parameter) transformer encoders, performs close to chance level (~50%) on PAWS, which specifically tests whether a model can distinguish sentence pairs with high lexical overlap but different meaning due to word order (e.g., "The cat chased the mouse" vs. "The mouse chased the cat").

Extensive experimentation (data rebalancing, focal loss, hard-negative augmentation, knowledge distillation from larger sentence-embedding teachers, and a separate bi-encoder + contrastive-loss architecture with anchor-matched hard-negative mining) was unable to move PAWS performance meaningfully above chance level. This appears to be a capacity-bound limitation of shallow (2-layer, small hidden-size) transformer encoders rather than a data or training-strategy issue — larger models (BERT-base and above) trained on PAWS reach 85–94% on the same benchmark.

Practical implication: this model is reliable for general semantic similarity / duplicate-question / paraphrase detection (QQP, MRPC, STS-B-style tasks), but should not be relied upon for adversarial or word-order-sensitive discrimination tasks.

Intended Use

• Lightweight semantic similarity scoring where model size/latency matters more than adversarial robustness
• Duplicate question detection, FAQ matching, basic paraphrase detection
• Not recommended for: legal/compliance-grade paraphrase detection, or any use case where word-order-reversal attacks are a realistic concern