mdbr-leaf-mt-prompt-safety

mdbr-leaf-mt-prompt-safety

A regression model for assessing English-language prompt safety. It evaluates user intent rather than just surface-level toxicity.

• Output: Float [0, 1] (0 = unsafe, 1 = safe).
• Primary Use: Guardrail for LLM applications.

Quickstart

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model_name = "agentlans/mdbr-leaf-mt-prompt-safety" 

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name).to(device)

text = "How do I blow up a balloon for a party?"
inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True).to(device)

model.eval()
with torch.no_grad():
    score = model(**inputs).logits.item()

print(f"Safety Score: {score:.4f}") # 0.8342

Examples

Training & Methodology

1. Pre-training: Masked Language Modeling (MLM) on prompt text.
2. Fine-tuning: Regression against safety scores from the agentlans/prompt-safety-scores dataset.

Hyperparameters

• Learning Rate: 5e-05
• Batch Size: 8
• Epochs: 20
• Optimizer: AdamW (Fused)
• Use Liger kernel

Performance (Regression)

Limitations

• No Rationale: Provides a score without explaining the specific safety violation.
• No Context: Evaluates single prompts; ignores conversational history.
• Fixed Policy: Safety criteria are fixed to training data and cannot be adjusted via prompts.
• Supportive Tool: Designed to complement, not replace, human moderation or broader content filters.

Licence

Apache 2.0