yalen-pii-ner-multilingual

Yalen PII-NER v2 — Multilingual PII & Entity Detector (DistilBERT · ONNX INT8)

Detects 9 classes of personal and financial data (names, organizations, locations, IBAN, card numbers, phone numbers, emails, dates) in French and English text, with a focus on medical documents. Developed by Yalen Labs.

v2 — retrained after a linguistic bias audit. Version 1 of this model was put through a three-phase bias audit: does it detect a Tunisian, Algerian, Senegalese or Kenyan patient name as well as a French one? The audit found a bias against Kenyan names and, above all, large failures on upper-case and lower-case names and on dates written in words. v2 was retrained from the base model to fix these issues, and meets all the audit's success criteria. Full reports, benchmark and evaluation code: YalenLabs/yalen-pii-ner-bias-audit.

What changed from v1

No regression on the 9 classes (internal validation, strict F1, v1 → v2): PER 65.3 → 97.2 %, ORG 70.7 → 81.6 %, LOC 81.8 → 90.0 %, MISC 48.7 → 74.5 %, IBAN 81.6 → 100 %, CARD 90.0 → 98.8 %, PHONE 89.7 → 99.2 %, EMAIL 98.9 → 100 %, DATE 68.3 → 100 %. On WikiNER, used to train both versions, they are on par (88.1 % vs 87.6 %).

v1 is still available with revision="v1".

Detected entities

The names in this card are fictitious examples.

Quick start

Transformers (PyTorch weights)

from transformers import pipeline

ner = pipeline("token-classification", model="YalenLabs/yalen-pii-ner-multilingual",
               aggregation_strategy="simple")
for e in ner("Patiente : MEDDEB Amal, née le 12 mars 1978, tél. +216 22 123 456."):
    print(e["entity_group"], e["word"], round(e["score"], 3))

ONNX Runtime (INT8, CPU)

import json, numpy as np, onnxruntime as ort
from huggingface_hub import snapshot_download
from transformers import AutoTokenizer

path = snapshot_download("YalenLabs/yalen-pii-ner-multilingual", allow_patterns=["*.onnx", "*.json", "vocab.txt"])
tok = AutoTokenizer.from_pretrained(path)
sess = ort.InferenceSession(f"{path}/model_quantized.onnx", providers=["CPUExecutionProvider"])
id2label = json.load(open(f"{path}/config.json"))["id2label"]

enc = tok("Patiente : MEDDEB Amal, née le 12 mars 1978.", return_tensors="np")
logits = sess.run(None, {"input_ids": enc["input_ids"], "attention_mask": enc["attention_mask"]})[0][0]
for t, l in zip(tok.convert_ids_to_tokens(enc["input_ids"][0]), logits.argmax(-1)):
    if id2label[str(l)] != "O":
        print(t, id2label[str(l)])

Training (v2)

Retrained from distilbert-base-multilingual-cased, 19 BIO labels (O + 9 × B/I), 3 epochs, lr 5e-5, batch 32, fp16, on one T4 GPU.

The bias-targeted data uses templates and surnames distinct from the audit benchmark, so the benchmark measures generalization. Contains data from ai4privacy/open-pii-masking-500k-ai4privacy, licensed under CC-BY-4.0.

Files

Limitations

• Evaluated on synthetic data. All audit benchmarks use clean, generated sentences. Scores close to 100 % show that v2 handles this kind of text, not that it will do as well on real documents (OCR, tables, typos). Validation on real anonymized documents is still needed.
• No class for social security numbers, addresses or medical record numbers. v1 sometimes masked social security numbers by mistake (tagging them as phone or card numbers); v2 is more precise and does so much less often. Use dedicated rules for these identifiers.
• Dates in words: 91 % detected, just above the 90 % target.
• Upper-case names with diaeresis ("AÏT AHMED") are sometimes detected only partially.
• Maximum input: 512 tokens; split long documents.
• MISC is inherited from WikiNER and can tag general named entities beyond personal data.

Intended use

Privacy compliance (GDPR), redaction of documents before sharing, data loss prevention, healthcare documents. The model produces detections for human-reviewed processes. It is not intended for profiling individuals or automated decisions about them.

License

MIT.