bert-base-banking77-student-pseudo-v1

Model Card for anant1213/bert-base-banking77-student-pseudo-v1 (BERT Student — Pseudo-Label Distilled)

Summary

This repository contains a BERT-base intent classifier fine-tuned on the BANKING77 dataset using a teacher pseudo-labeling distillation approach (Path 1). A fine-tuned BERT-large teacher generates predicted intent labels for each training example; the BERT-base student is then fine-tuned on these teacher-generated labels and evaluated on the original test set.

---

Model Details

Model Description

• Developed by: <ANANT SRIVASTAVA / anant1213 >
• Shared by: <ANANT SRIVASTAVA / anant1213 >
• Model type: BertForSequenceClassification (BERT encoder + classification head)
• Language(s): English
• License: Apache-2.0 (inherits from base BERT license) :contentReference[oaicite:0]{index=0}
• Finetuned from model: google-bert/bert-base-uncased :contentReference[oaicite:1]{index=1}
• Teacher model (used for pseudo-labeling): google-bert/bert-large-uncased fine-tuned on BANKING77 :contentReference[oaicite:2]{index=2}

Model Sources

• Base student model: google-bert/bert-base-uncased :contentReference[oaicite:3]{index=3}
• Base teacher model: google-bert/bert-large-uncased :contentReference[oaicite:4]{index=4}
• Dataset: PolyAI/banking77 :contentReference[oaicite:5]{index=5}

---

Uses

Direct Use

• Classifying short banking-domain queries into one of 77 intent labels (e.g., “card payment reversed”, “balance not updated”, etc.). :contentReference[oaicite:6]{index=6}

Downstream Use

• As a lightweight intent router for customer support workflows
• As a feature extractor (CLS embedding) for downstream classifiers (requires custom code)
• As a student initialization for further fine-tuning on a proprietary intent dataset (recommended)

Out-of-Scope Use

• General-domain intent classification outside banking
• Long-document classification (model trained on short queries)
• High-stakes decision making without human verification

---

Bias, Risks, and Limitations

• Domain limitation: Trained on banking queries; performance may degrade on other domains. :contentReference[oaicite:7]{index=7}
• Teacher bias transfer: Student inherits systematic teacher errors since training labels are teacher predictions (pseudo labels).
• Label noise: If the teacher is imperfect, pseudo labels introduce noise that can cap student performance.
• No calibrated confidence guarantee: Softmax confidence may not reflect true correctness; calibrate if used in production.

Recommendations

• If deploying in production, validate on in-domain data and monitor intent confusion pairs.
• Consider adding a fallback intent (e.g., “unknown”) or thresholding for low-confidence predictions.

---

How to Get Started with the Model

Inference Example

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

repo_id = "anant1213/bert-base-banking77-student-pseudo-v1"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForSequenceClassification.from_pretrained(repo_id)

text = "I was charged twice for the same card payment"
inputs = tokenizer(text, return_tensors="pt", truncation=True)

with torch.no_grad():
    logits = model(**inputs).logits
pred_id = int(torch.argmax(logits, dim=-1))
print("Predicted label id:", pred_id)