roblox-pii-classifier-v2

Roblox PII Classifier v2

Overview

We present Roblox/roblox-pii-classifier-v2, an upgraded, context-aware PII (Personally Identifiable Information) detection model. Built on the XLM-RoBERTa-Large architecture, v2 is designed to identify attempts to share or solicit personal information across complex, multi-user conversations.

Unlike its predecessor and other NER models, v2 evaluates text within its surrounding conversational context, allowing it to detect sophisticated, adversarial bypass attempts (e.g., collaborative PII exchange, phonetic spelling, implicit references) while drastically reducing false positives. It is intended for use in casual conversations, ensuring freedom of speech while also maintaining safety.

What’s New in v2?

• Conversational Context Integration: Evaluates target messages within the broader multi-user chat context, eliminating ambiguity inherent in isolated messages.
• Massive Multilingual Expansion: Broadened synthetic data generation and fine-tuning capabilities to expand language coverage from 17 to 189 languages.
• Automated Red-Teaming & Clustering: Applied automated adversarial red-teaming and cluster visualization to systematically surface, isolate, and patch model vulnerabilities.
• Significant Performance Leap: Overall F1 score increased from 63.41% to 90.52% on our refreshed internal evaluation set, with a f1 score increase from 64.74% to 88.78% on the recently released roblox-pii-safety-for-chat-benchmark.

Model Architecture & Classification Scope

Input Format: A single pre-formatted string containing a fixed instruction prefix followed by the target message and its surrounding multi-turn chat history. Speakers are anonymized — the speaker under evaluation is always t, and all other participants are mapped to s1, s2, … in order of first appearance. Turns are joined with </s>:

Instruct: In the following chat messages from target speaker t and possibly other speakers s1, s2, etc., detect abuse by speaker t.
Query:

t: {text} </s> s1: {text} </s> t: {text}

Tokenized with the XLM-RoBERTa SentencePiece tokenizer to a fixed 512 tokens, padding="max_length", truncation=True, and — critically — truncation_side="left", so that when history overflows the window the oldest turns are dropped and the latest target message is retained.

Outputs: The model performs multi-label classification scores obtained by applying an element-wise sigmoid to the logits across three PII categories, emitted in this fixed index order:

Benchmark Comparisons

The table below illustrates best f1 performance across internal and open-source evaluation benchmarks (full conversational data used when possible):

* These models all were supplied with only the target-user text as other speaker texts are meant to be supplemental contextual information. roblox-pii-classifier-v2 performs best when supplied with conversational context, however it still outperforms roblox-pii-classifier when supplied only with target-user text.

Usage

Install the inference dependencies:

pip install -r requirements.txt

For a single message, pass the raw text. inference.py treats it as one turn from the target speaker t and adds the required instruction prefix automatically:

python inference.py --model-path . \
  --text "add me on Discord, my username is skyfox_4821"

For conversational inference, create a UTF-8 JSON file containing a list of turns. The speaker whose behavior should be classified must be named t. Other speaker names may be arbitrary; the formatter anonymizes them to s1, s2, and so on in order of first appearance.

cat > conversation.json <<'JSON'
[
  {"speaker": "alice", "text": "how can I contact you?"},
  {"speaker": "t", "text": "add me on Discord, my username is skyfox_4821"},
  {"speaker": "alice", "text": "okay, got it"}
]
JSON

python inference.py --model-path . --input-file conversation.json