bert-phishing-detector

BERT Email Phishing Detector

Fine-tuned bert-base-uncased for binary classification of email/message text as phishing or legitimate.

Labels

Intended use

• Research and education on phishing detection.
• A component / baseline in a larger email-security pipeline.

Not intended as a standalone production email security gateway. Always keep a human in the loop for high-stakes decisions, and combine with URL reputation, sender authentication (SPF/DKIM/DMARC) and other signals.

How to use

from transformers import pipeline

clf = pipeline("text-classification", model="<your-username>/bert-phishing-detector")
clf("Verify your account now at http://secure-login.example.ru")
# [{'label': 'phishing', 'score': 0.99}]

Training

• Base model: bert-base-uncased
• Objective: binary sequence classification (cross-entropy)
• Max sequence length: 256 tokens
• Optimizer / LR: AdamW, 2e-5, weight decay 0.01, 6% warmup
• Epochs: 3 (best checkpoint by validation F1)

Training code: src/train.py in the project repository.

Evaluation

Fine-tuned for 3 epochs on ~65.7k emails and evaluated on an 8,207-email held-out test split:

These scores are very high partly because the training data (the Kaggle phishing-email collection) is cleanly separable. Expect lower performance on noisier, real-world email — validate on your own data before relying on it.

Limitations & biases

• Distribution shift is the big one. The training corpus is heavily preprocessed (lowercased, punctuation and stop-words removed) and the legitimate emails skew toward early-2000s corporate mail (Enron). The model scores ~99% on held-out data from this same distribution, but can misclassify raw, modern, or very short emails (e.g. a casual "Hi team, notes attached" message may be flagged as phishing). For best results, feed it text preprocessed the same way as training, and treat the headline metrics as in-distribution — not a promise of real-world accuracy.
• Trained primarily on English text; performance on other languages is untested.
• Phishing tactics evolve quickly; the model can go stale and should be periodically retrained on fresh data.
• May produce false positives on legitimate marketing emails that share surface features with phishing (urgency, links, "verify" language).
• The bundled data/sample_emails.csv is a tiny illustrative sample. Train on a large, representative corpus before relying on any metric.

Data

Bring your own labeled corpus with a text column and an integer label column (0 = legitimate, 1 = phishing). Commonly used public sources include the Nazario phishing corpus, the Enron email dataset (legitimate mail), SpamAssassin public corpus, and various Kaggle phishing-email datasets. Check each dataset's license before redistribution.

Ethical considerations

This model classifies text and can make mistakes. Do not use it to automatically delete mail or take irreversible actions without human review.