motivbot-multilingual-mdeberta-v3

MotivBot Multilingual Mental Health Classifier — mDeBERTa v3 (Best Fold)

This model uses mDeBERTa‑v3‑base with CLS+mean pooling and structured features (Age, Gender, PHQ‑9, RSES).
It is the best single fold ({best_fold}) from a 5‑fold ensemble trained with:

• LLRD (backbone=3e‑5, heads=5e‑4)
• Focal Loss + label smoothing
• Dropout 0.25
• Weight decay 0.02
• Lightweight augmentation (random word drop)
• PyTorch SWA (AveragedModel + update_bn)
• 35 epochs, patience 7, warmup 20%

Validation ternary macro‑F1: {best_f1:.4f}

Load the model

import torch
import json
from transformers import AutoTokenizer
from modeling import MentalHealthClassifierV6

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

with open("model_config.json") as f:
    config = json.load(f)
with open("preprocess_stats.json") as f:
    stats = json.load(f)

model = MentalHealthClassifierV6(model_name=config["model_name"])
model.load_state_dict(torch.load("pytorch_model.bin", map_location=device))
model.to(device).eval()
tokenizer = AutoTokenizer.from_pretrained(".")

# Normalise structured features using stats:
age_norm = (age - stats['num_mean']['Age']) / stats['num_std']['Age']
phq9_norm = (phq9 - stats['num_mean']['PHQ9']) / stats['num_std']['PHQ9']
rses_norm = (rses - stats['num_mean']['RSES']) / stats['num_std']['RSES']
gender_enc = {{'male':0, 'female':1, 'other':2}}.get(gender.lower(), 2)
gender_norm = (gender_enc - stats['g_mean']) / stats['g_std']
struct = torch.tensor([[age_norm, phq9_norm, rses_norm, gender_norm]], dtype=torch.float32).to(device)

inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=config["max_len"]).to(device)
with torch.no_grad():
    logits = model(inputs["input_ids"], inputs["attention_mask"], struct)
probs = [torch.softmax(l, dim=1).cpu().numpy() for l in logits]