modernbert-banking77

ModernBERT fine-tuned on Banking77

Fine-tune of answerdotai/ModernBERT-base for 77-class customer-intent classification.

Usage

from transformers import pipeline
clf = pipeline("text-classification", model="Ahmed167/modernbert-banking77", top_k=5)
clf("My card hasn't arrived yet, what should I do?")

Training

• Base model: answerdotai/ModernBERT-base (149M params, encoder-only, 8k context)
• Dataset: PolyAI/banking77 (~13k train / 3k test, 77 classes)
• Recipe: full fine-tune, lr=5e-5 cosine schedule with 10% warmup, 4 epochs, bf16, weight-decay=0.01, label-smoothing=0.1, batch size 32
• Validation split: 10% of train, stratified by label
• Metric for best model: macro F1

Calibration

Logits are saved as-is. To get calibrated probabilities, divide logits by the fitted temperature (reported in eval_results.json in the source repo) before softmax. The fitted T value is typically slightly > 1, meaning the raw model is mildly overconfident.

Source

GitHub: AhmedMostafa167/Projects/04-Deep-Learning/text-classification-modernbert