nym-pii-multilingual-small

nym-pii-multilingual-small

Compact multilingual PII token-classification model for the nym anonymization CLI — a 16-layer, vocabulary-pruned, distilled student of Wismut/nym-pii-multilingual (mmBERT architecture). 429 MB fp32 / 133 MB compressed / 104 MB edge.

40 entity types / 81 BIO labels across ~23 languages and 6 scripts (Latin, Cyrillic, CJK, Korean, Arabic, Devanagari), trained with OCR-style corruption for robustness to scanned/mangled text, and with structured-document formats (JSON, CSV, tables, key-value, EDI) so field names and delimiters aren't mistaken for PII.

Files

Compression is never free by assumption — every variant above was re-run on the full benchmark battery (v2's int8 shipped with a "99.9% agreement" claim that turned out to hide an 8-point OOD loss; that mistake is not repeated here).

Benchmarks

Same harness for every row (span-level, label-agnostic; OOD sets never trained on). v2 rows are the previously published artifacts, re-measured today:

On the independent REDACT benchmark (25 languages / 9 scripts / 51 PII types, type-matched partial micro-F1, their harness): this model 0.507 — above the OpenAI Privacy Filter (0.512 → base tier 0.515 beats it, small ties), far above GLiNER-multi (0.320) and Presidio (0.195), below GPT-4.1 (0.597) and Claude Sonnet 4.6 (0.636).

Honest trade vs OpenMed: on curated real text their models win the F1 column — the 44M SuperClinical-Small by +1.7 over this model, their multilingual 279M mSuperClinical by +5.0. Off-distribution the picture inverts hard: this model is +23.1 ai4 over the 44M and +12.7 ai4 / +11.4 non-Latin over their multilingual best, at an 8× smaller compressed artifact (133 MB vs ~1.1 GB). Pick by whether your text looks like a clinical corpus or like the wild.

How it was trained

1. Init: jhu-clsp/mmBERT-small depth-pruned 22→16 layers.
2. Synthetic (724k): exactly-labeled data from Wismut/nym-pii-multilingual-data, including structured-format templates (JSON/CSV/table/key-value/EDI).
3. Real text (77.5k): Wikipedia passages auto-labeled by a large LLM teacher (gemma-4-26b); weak O tokens masked from the loss.
4. Knowledge distillation from the retrained mmBERT-base teacher (--distill-from, KL on all attended tokens, α=0.5, T=2) — sharpens calibration, worth ~+3 ai4.
5. Recall tilt (--o-weight 0.3): down-weights the O class; combined with KD it is super-additive (tilt alone costs precision; KD repairs it).
6. Compression: vocab prune (0.999 coverage) → embedding-only int8 → weight-only fp16 body. Each step individually re-benchmarked.

Pipeline: scripts/ (datagen/, prune/, train_ner.py, export_onnx.py).

Versions

• v3 (current) — retrained-teacher KD + recall tilt; corrected ONNX export (v2's export env silently ran a different transformers version whose ModernBERT forward differs — the shipped v2 artifact lost ~3.5 ai4 vs its own checkpoint; v3 exports are gate-verified torch-vs-ONNX ≤1e-3 at 10 lengths). Same-harness gains over shipped v2: +3.5 F1, +12.1 ai4, +3.2 non-Latin; the int8 tier gains +6.4 F1 / +20 ai4 over v2-int8.
• v2 — LLM-teacher-labeled real text + structured-format synthetic.
• v1 — synthetic + mmBERT-teacher-labeled real text.

Use with nym

[ner]
enabled = true
backend = "tokens"
token_model = "Wismut/nym-pii-multilingual-small"        # fp32, accuracy-first
# token_model = "Wismut/nym-pii-multilingual-small/int8" # 133 MB, near-lossless
# token_model = "Wismut/nym-pii-multilingual-small/edge-int8" # 104 MB, edge
threshold = 0.25
# recall_first = true   # flag on total entity mass (1-P(O)): +5-9 char-recall
#                       # at ~-3 precision. Right mode for redaction; keep off
#                       # when label accuracy matters more than coverage.

Limitations

• The pruned tokenizer + ONNX layout targets nym / onnxruntime; for transformers fine-tuning use the teacher repo (Wismut/nym-pii-multilingual).
• Trained partly on LLM-labeled Wikipedia text (CC-BY-SA source; not redistributed here). Those labels are machine-generated.
• Recall-leaning by design; expect over-flagging on ambiguous terms.
• Reference-style IDs (ticket/case numbers) are the weakest class on out-of-taxonomy benchmarks — pair with nym's regex patterns, which is how the full pipeline runs anyway.
• No claim of HIPAA/GDPR compliance by itself; it is a detection component.

License

MIT (inherited from mmBERT-small). The synthetic training data contains no real personal information; the real-text training corpus is not distributed with this model.