motivbot-multilingual-mdeberta-v4

MotivBot Multilingual Mental Health Classifier — mDeBERTa v4 (Best Fold)

This model uses mDeBERTa‑v3‑base with CLS+mean pooling and structured features (Age, Gender, PHQ‑9, RSES).
It is the best single fold (4) from a 5‑fold ensemble trained with:

• Longer context (384 tokens)
• LLRD (backbone=5e‑5, heads=5e‑4)
• Focal Loss + label smoothing
• Dropout 0.3
• Weight decay 0.02
• Augmentation: word drop + synonym replacement
• PyTorch SWA (AveragedModel + update_bn)
• 35 epochs, patience 7, warmup 25%, Low class weight boosted ×2

Validation ternary macro‑F1: 0.7166

Load the model

import torch
import json
from transformers import AutoTokenizer
from modeling import MentalHealthClassifierV6

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

with open("model_config.json") as f:
    config = json.load(f)
with open("preprocess_stats.json") as f:
    stats = json.load(f)

model = MentalHealthClassifierV6(model_name=config["model_name"])
model.load_state_dict(torch.load("pytorch_model.bin", map_location=device))
model.to(device).eval()
tokenizer = AutoTokenizer.from_pretrained(".")

# Normalise structured features using stats:
age_norm = (age - stats['num_mean']['Age']) / stats['num_std']['Age']
phq9_norm = (phq9 - stats['num_mean']['PHQ9']) / stats['num_std']['PHQ9']
rses_norm = (rses - stats['num_mean']['RSES']) / stats['num_std']['RSES']
gender_enc = {'male':0, 'female':1, 'other':2}.get(gender.lower(), 2)
gender_norm = (gender_enc - stats['g_mean']) / stats['g_std']
struct = torch.tensor([[age_norm, phq9_norm, rses_norm, gender_norm]], dtype=torch.float32).to(device)

inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=config["max_len"]).to(device)
with torch.no_grad():
    logits = model(inputs["input_ids"], inputs["attention_mask"], struct)
probs = [torch.softmax(l, dim=1).cpu().numpy() for l in logits]