Mila-Suicide-Prevention-Output-Guardrail

Model Card for Mila Suicide Prevention Output Guardrail

A small classifier that flags LLM responses that enable, encourage, or endorse suicide or self-harm, in English and French.

Beta release. Not intended for standalone production use. Deploy with human oversight and additional safety layers.

Model Details

The model reads a single assistant response and predicts whether it violates our suicide and self-harm policy. It is one layer in a defense-in-depth stack: not an input guardrail, not a general harm classifier, and not a standalone moderation system.

• Developed by: Mila and Mila AI Safety Studio
• Point of contact: Mila
• Version: 0.1.0-beta.3
• Base model: mmBERT-base
• Architecture: ModernBertForSequenceClassification, single-logit head
• Languages: English, French
• Input: one assistant response, up to 512 tokens (longer text is truncated)
• Output: p(unsafe) in [0, 1]; label 1 if p(unsafe) ≥ τ, else 0
• Default threshold: τ = 0.5

Usage

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

repo_id = "mila-ai4h/Mila-Suicide-Prevention-Output-Guardrail"
THRESHOLD = 0.5  # choose an operating point from the Performance tables

tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForSequenceClassification.from_pretrained(repo_id)
model.eval()

texts = [
    "I hear how heavy this feels. Please call or text 988 if you are in crisis.",
    "Je comprends que c'est difficile. Tu peux appeler le 9-8-8 si tu es en crise.",
]

inputs = tokenizer(texts, padding=True, truncation=True, max_length=512, return_tensors="pt")

with torch.no_grad():
    p_unsafe = torch.sigmoid(model(**inputs).logits.squeeze(-1))

for text, p in zip(texts, p_unsafe.tolist()):
    label = "unsafe" if p >= THRESHOLD else "safe"
    print(f"{label} (p_unsafe={p:.4f})  {text}")

Expected output:

safe (p_unsafe=0.0370)  I hear how heavy this feels. Please call or text 988 if you are in crisis.
safe (p_unsafe=0.0305)  Je comprends que c'est difficile. Tu peux appeler le 9-8-8 si tu es en crise.

Policy

Based on the "Suicide and Self-Harm" category of MLCommons AILuminate.

Labeling rules:

• Commission-based: a response is labeled for what it says. Missing crisis resources, disclaimers, or a poor tone do not make it 1.
• Ambiguity: benign intent is assumed; truly ambiguous responses are 0.
• Knowledge: generic, non-actionable information is acceptable; information beyond common knowledge, or synthesized into a plan of action, is not.
• Narrative: characters in fiction may say what the assistant itself may not.
• Education: otherwise harmful content may be acceptable in an educational context presented with care.

Uses

Intended use:

• Output moderation at response time in English or French chat systems, as one layer alongside other safeguards.
• Research and benchmarking of lightweight, domain-specific guardrails.

The model is small enough to run on every response, where larger policy models may be too slow or costly.

Mila does not endorse any use that seeks to bypass safeguards, operationalize harmful behavior, or weaken human oversight in high-stakes contexts.

Out-of-scope use:

• Moderating user inputs.
• General safety filtering or other harm categories.
• Clinical diagnosis, triage, or crisis intervention.
• Decisions in high-stakes or life-safety contexts without human review.
• Languages other than English and French.

Performance

Internal evaluation set: 3,393 English responses, each with a French translation. The evaluation set is private and will not be released.

• Human (n = 413): labeled by 3 annotators (Fleiss κ = 0.842).
• Extended (n = 2,983): labeled by an LLM-judge panel (agreement with human labels from above, Fleiss κ = 0.985).

For the recall operating points, τ is the threshold that reaches the target recall, rounded to 2 decimals.

English (n = 3,393). AUPRC: 0.1.0-beta.2 = 0.970, 0.1.0-beta.3 = 0.997.

French (n = 3,393). AUPRC: 0.1.0-beta.3 = 0.996.

Note: 0.1.0-beta.2 was trained on English only, so French is not reported. Its previously published results (precision 0.908, recall 0.802 at τ = 0.5) used an older benchmark and are not comparable.

Limitations

• False negatives: subtle, adversarial, or out-of-distribution content may be missed. A 0 does not guarantee safety.
• False positives: supportive responses that discuss self-harm may be flagged.
• No conversation context: harm that depends on earlier turns (e.g., a neutral answer to a user who stated intent) is invisible to the model.
• No explanations: the model outputs a score, not a rationale or subcategory.
• Training data: responses are LLM-generated and labels come from LLM judges; real-world distribution shift is expected.
• French: French training and evaluation data are machine translations; performance on native French text is not measured.
• Evaluation: the evaluation set was extended on cases where LLM judges agree, so it under-represents borderline cases.

Planned: multi-turn context, broader edge-case coverage, and improved multilingual support.

Training

Training follows a policy-driven approach inspired by constitutional classifiers (Sharma et al., 2025): a natural-language policy defines which responses are acceptable, and that policy is used to label the data that supervises a small classifier. The constitution can be found in the artifacts, in constitution.md. The policy shapes the data and labels only. At inference, the model sees the response text alone.

The training data is private and will not be released.

Change Log

0.1.0-beta.3

• Refined the policy to cover additional edge cases.
• Updated the internal evaluation dataset, now in two splits:
  • Human (n = 413): labeled by 3 annotators (Fleiss κ = 0.842).
  • Extended (n = 2,983): labeled by an LLM-judge panel (agreement with human labels from above, Fleiss κ = 0.985).
• Relabeled the training dataset under the updated policy.
• Changed the base model from bert-base-uncased to mmBERT-base.
• Added French support.

0.1.0-beta.2

• Updated the policy to align with MLCommons AILuminate "Suicide and Self-Harm" category.
• Generated a new synthetic training dataset.
• Trained bert-base-uncased on the new dataset.
• Updated our internal evaluation dataset to align with the new policy and validated the new model on it.

Citation

@misc{mila-suicide-prevention-output-guardrail-0.1.0-beta.3-2026,
  title={Mila Suicide Prevention Output Guardrail (0.1.0-beta.3)},
  author={Mila AI4H},
  year={2026},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/mila-ai4h/Mila-Suicide-Prevention-Output-Guardrail}}
}