mbert-multilingual-legal-ner-pseudonymization

Multilingual Legal NER — PII Pseudonymization (PL · DE · EN)

A multilingual token-classification (NER) model for detecting personally identifiable information (PII) in legal and administrative text, covering Polish, German and English in a single model.

This was Lexedit's original production PII detector (internally ner-legal-v5). It is published for the case that motivates it: one model for mixed-language EU document sets, where routing per language is impractical.

Pseudonymization, not anonymization

The distinction is legal, not stylistic, and it matters under GDPR:

• Pseudonymization (art. 4(5) GDPR) — identifiers are replaced with pseudonyms, and the mapping is kept somewhere so the process is reversible. Pseudonymized data is still personal data and stays in GDPR scope.
• Anonymization — irreversible; the data subject can no longer be identified by anyone, by any reasonably likely means. Truly anonymized data falls outside GDPR (recital 26).

This model does neither on its own — it detects spans. What you do with those spans decides which regime you are in. If you replace Jan Kowalski with [PER_a1b2] and retain a mapping (the typical setup, and the one Lexedit uses), you have pseudonymized. That is a security and data-minimisation measure, and a legitimate one — but do not describe the result as anonymous data, and do not assume it is exempt from GDPR obligations.

Reaching genuine anonymization requires an irreversible transform, discarding the mapping, and handling indirect re-identification (quasi-identifiers, singling out via rare attribute combinations) — none of which a NER model addresses.

Which model should you use?

Your documents	Use
Polish only, clean/digital	lexedit/herbert-polish-legal-ner — substantially better
Polish only, scanned/OCR'd	lexedit/herbert-polish-legal-ner-ocr
German / English / mixed-language	this model (the only one that covers them)

On Polish this model is clearly weaker than the HerBERT siblings — a 15.6 % vs 10.1 % identity-level leak on the same evaluation set, in the same recall-first configuration. Do not pick it for Polish-only work. Its reason to exist is language coverage, not Polish accuracy.

Intended use is defensive: flagging PII so it can be masked before a document is shared or processed. It is not a guarantee that every identifier is caught — see Limitations.

Labels (29, BIO scheme)

PER (person) · ORG (organisation) · LOC (private address/location) · LOC_PUB (public place: city, country) · DATE · MONEY · EMAIL · PHONE · ID (national id / case / document number) · IBAN · DIAGNOSIS · HEALTH_FACILITY · MEDICAL_ID · WATERMARK, each as B-… / I-…, plus O.

The model distinguishes private locations (LOC, mask these) from public ones (LOC_PUB, usually safe to keep), and treats DATE / MONEY as non-masked by default. This distinction holds across all three languages.

WATERMARK is a legacy label from the original training corpus (a provenance marker present in part of the synthetic data). It is not useful for pseudonymization — ignore it.

Quick start (ONNX, no PyTorch required)

pip install onnxruntime transformers numpy
python examples/inference_onnx.py

import json, numpy as np, onnxruntime as ort
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("lexedit/mbert-multilingual-legal-ner-pseudonymization")
sess = ort.InferenceSession("onnx/model_quantized.onnx")
enc = tok("Der Beklagte Thomas Müller, wohnhaft Hauptstraße 12 in Berlin.",
          return_offsets_mapping=True, return_tensors="np")
enc.pop("offset_mapping")
feeds = {i.name: enc[i.name].astype(np.int64) for i in sess.get_inputs()}
logits = sess.run(None, feeds)[0]          # (1, seq, 29)
# argmax per token -> map ids via config.id2label -> group B-/I- with offset_mapping

See examples/inference_onnx.py for span grouping that works out of the box.

Actual output of that script:

[pl] Pozwany Jan Kowalski, zam. ul. Słoneczna 5 w Krakowie, PESEL 02070803628, e-mail [email protected].
    PER              'Jan Kowalski'
    LOC              'ul. Słoneczna 5'
    LOC_PUB          'Krakowie'
    ID               '02070803628'
    EMAIL            '[email protected]'

[de] Der Beklagte Thomas Müller, wohnhaft Hauptstraße 12 in Berlin, IBAN DE89370400440532013000.
    PER              'Thomas Müller'
    LOC              'Hauptstraße 12'
    LOC_PUB          'Berlin'
    IBAN             'DE89370400440532013000'

[en] The defendant John Smith, residing at 5 Sunny Road in London, e-mail [email protected].
    PER              'John Smith'
    LOC              '5 Sunny Road'
    LOC_PUB          'London'
    EMAIL            '[email protected]'

Runs in the browser

The quantized ONNX (~174 MB, int8) loads client-side via onnxruntime-web or transformers.js (WASM), so the document never leaves the user's device — the point of the exercise for sensitive legal text. It also runs anywhere ONNX Runtime does (Python, Node, server, mobile). Context window is 512 subword tokens; chunk longer documents with overlap and merge spans across chunk boundaries.

Evaluation — read this carefully

Polish is the only language with a real evaluation. Identity-level leak rate = a person counts as leaked if any mention of them is missed. Internal set of 50 real Polish legal documents (179 persons):

Token-level F1 on the held-out synthetic test split ≈ 0.93 (P 0.916 / R 0.942) — this number is inflated; synthetic held-out data substantially overstates real-document performance. Trust the leak-rate column.

⚠️ German and English are unevaluated

There is no DE or EN evaluation set. Both languages were present in training and the model demonstrably produces correct spans on them (see the sample output above), but no measured accuracy, recall or leak rate exists for German or English. Treat DE/EN capability as plausible and untested. If you deploy on those languages, build your own evaluation set first — do not assume the Polish numbers transfer.

Recommended production setup

This model is recall-first and is one layer of a pipeline, not the whole solution. Pair it with:

1. A recall-first threshold for PER (flip a token to PER when the summed PER probability ≥ ~0.2, even if it is not the arg-max). This alone cut the Polish leak rate from 19.0 % to 15.6 %.
2. A deterministic post-pass — snap spans to whole words, merge hyphenated surnames, propagate a detected surname to its other inflected mentions.
3. Checksum-validated regex for structured PII (PESEL, NIP, REGON, IBAN, …) — the model is not a reliable detector of well-formed identifiers on its own.
4. Human review for high-stakes use.

Training data

Fine-tuned on ~3 500 LLM-generated synthetic legal and administrative documents in Polish, German and English, with programmatically generated labelled PII (names, addresses, national identifiers, IBANs, contact details) drawn from per-language pools.

The corpus is entirely synthetic — no real personal data was used, and none is distributed with this model. Because this is a token-classification model (one label per input token; it cannot generate text), the weights do not reproduce or expose training documents.

The synthetic nature of the training data is also this model's main weakness: it was tuned on generated documents whose structure is more regular than real filings.

Limitations

• Polish accuracy is beaten by the HerBERT siblings — use those for Polish-only workloads.
• German and English are untested (see above). No metrics exist.
• Not a guarantee. A residual leak rate remains (≈15–19 % identity-level on real Polish documents); always combine with the post-pass, checksum regex and human review for high-stakes use.
• Scanned / OCR'd text degrades results — garbled names are frequently missed.
• All-caps names (common in signature blocks and scan headers) are a known weak spot of this model's WordPiece tokenizer.
• 512-token context; long documents must be chunked.
• Small evaluation set; numbers are indicative, not a benchmark.
• Not legal advice; not a substitute for a privacy/compliance review.

License — CC BY 4.0 (attribution required, commercial use allowed)

Released under Creative Commons Attribution 4.0. You may use, modify and redistribute this model — including in commercial products — provided you give appropriate credit.

Suggested attribution:

Multilingual legal NER / PII pseudonymization model by lexedit (https://lexedit.ai), licensed CC BY 4.0, fine-tuned from Davlan/bert-base-multilingual-cased-ner-hrl.

This model is a derivative of Davlan/bert-base-multilingual-cased-ner-hrl (Apache-2.0), itself derived from bert-base-multilingual-cased (Apache-2.0). Please retain attribution to the base models as well.