therapy-safety-router-v3

Therapy Safety Router v3 (LoRA)

Safety layer for wellbeing chatbots. Classifies an incoming user message; never replies to the user, never diagnoses.

message -> {risk, request, injection_attempt} -> the host system picks an action.

Decide from the probability, not the generated token

Prefill {"risk":" and read the distribution over the next token, then compare against a threshold. Rounding at 0.5 hides real signal: five of eight rows this model appeared "blind" to were in fact seen, at p between 0.20 and 0.56.

The threshold is a runtime dial, not a model constant.

Training

Qwen2.5-1.5B-Instruct + LoRA r=16, alpha=32, all linear modules, lr=2e-4, 3 epochs, loss on assistant response only. 526 synthetic rows, labelled by a 3-model jury with planted control rows.

Limits

Not a medical device - one layer among several, human escalation mandatory. 100% synthetic data, English only. The crisis rubric has not been reviewed by a clinician. Proving recall 0.98 would need ~150 risky eval rows; this eval has 49.