general-pii-detector

General PII Detector — 23 labels, document level

Give it text of any length; it returns which of 23 categories of personal or sensitive information are present. Long inputs are windowed internally, so callers never deal with a context limit.

Model Details

Model Description

A multi-label document classifier: a multilingual encoder with one linear head producing 23 independent probabilities, each compared against its own threshold.

This is a presence detector, not an extractor. It reports that a phone number is somewhere in the text; it does not say where, and it does not return the value.

• Model type: multi-label text classification (encoder + linear head)
• Languages: multilingual; evaluated on Chinese, English and Indonesian
• Base model: jhu-clsp/mmBERT-base (ModernBERT architecture)
• Parameters: 307M, of which 196.6M are embeddings
• Licence: Apache 2.0, inherited from the base encoder

Measured on real traffic

Training, calibration and evaluation all use real API traffic — captured HTTP request and response bodies, labelled document by document. No synthetic data was used for any of the numbers below. Splits are grouped by API endpoint, never by row, so near-duplicate documents from the same endpoint cannot straddle training and evaluation.

Over the 14 labels with enough positives and negatives to measure, on 4,359 held-out real documents:

The strongest labels, with their real precision and recall at the shipped threshold:

The full 23-label table, including the weak labels and what the pretraining stage bought each of them, is under Evaluation.

Model Sources

• Windowing and aggregation library: sliding-window-transformers

Uses

Direct Use

Flagging which categories of personal information appear in a document, for data-inventory, routing or triage work: deciding which API endpoints handle sensitive data, which logs need stricter retention, which records need review.

Out-of-Scope Use

• Not an extractor. No spans, no values, no redaction. Pair it with rules or a token-level model if you need the value itself.
• Not a compliance control. Two labels are broken and three were never evaluated. Do not make it the only safeguard for a regulatory obligation.
• Not calibrated outside API traffic. See Bias, Risks and Limitations.

Bias, Risks and Limitations

What each label is actually worth. Two things differ and are easy to conflate: whether a label has measured accuracy (enough positives in the evaluation split) and whether its threshold was fitted on the separate calibration split. 14 labels have the first, 13 have the second. Where a threshold was not fitted, the label keeps the 0.5 default — which is unoptimised, not automatically bad. Measured at that default:

So: two labels are broken (biometric_id, house_number), three are entirely unmeasured (passport_number, social_security_number, vehicle_plate — the evaluation split contains no positive example of them), password recalls half of what it should, and the remaining 17 work. Every one of these counts is small, so treat the numbers as indicative rather than tight.

Labels suppress one another on dense documents. A card number alone scores 1.000; in a document that also carries a name, a phone, an address, an e-mail and an IP it falls to 0.008. The step-by-step measurement is under Illustrative examples below. A label reported at 0.82 AP can be near-zero on a PII-dense document, so aggregate accuracy is not a safe proxy for dense traffic.

Distribution. Trained and calibrated on HTTP request and response bodies from API traffic. On other shapes (CSV, SQL dumps, logs, e-mail, prose) the high-frequency labels transfer in spot checks, but the rare ones do not and no threshold is calibrated for them.

Isolated values are missed. A bare phone number on its own line scores near zero; the same number inside a sentence scores 1.000. The model learned "value plus context", not pattern matching.

Cross-window association is impossible. Aggregation across windows is a disjunction, so evidence more than ~510 tokens apart cannot be combined. Every label here is locally decidable, so this costs nothing in practice — but a label that needed "A and B hold far apart" would be out of reach for this architecture.

Recommendations

Measure per-label recall on your own data before relying on any label, and do it on documents that are as dense as your real traffic. If you adopt this outside API traffic, refit the thresholds on a calibration sample of your own distribution; that is far cheaper than retraining and recovers most of the loss.

How to Get Started with the Model

from transformers import AutoModel, AutoTokenizer

REPO = "Zeo6/general-pii-detector"
model = AutoModel.from_pretrained(REPO, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(REPO)

result = model.detect(text, tokenizer=tokenizer)
# {"labels": ["phone_number", "person_name"], "windows": 1, "scores": {...}}

text is a plain string of any length. That is the whole interface — no chunking, no batching of windows, no threshold handling on your side. Pass a list to score several documents at once.

If you would rather not enable trust_remote_code, download the repo and import the single file directly; it needs nothing beyond torch and transformers:

hf download Zeo6/general-pii-detector --local-dir ./general-pii-detector

import sys
sys.path.insert(0, "./general-pii-detector")
from modeling_general_pii import GeneralPIIDetector

detector = GeneralPIIDetector.from_pretrained("./general-pii-detector")
detector.detect("...")                      # the tokenizer loads from the same directory
detector.detect(["doc one", "doc two"])     # a list in, a list out

Training Details

Training Procedure

Three stages. The base encoder was first fine-tuned on labelled traffic; that checkpoint then received domain-adaptive masked-LM pretraining on unlabelled text of the same distribution (holdout MLM loss 13.1 → 0.74); the adapted encoder was then fine-tuned with all 22 layers unfrozen, for three random seeds, and the three resulting checkpoints were averaged uniformly.

Fine-tuning uses multiple-instance learning: a document is a bag of windows, each label back-propagates through its own single highest-scoring window, and label sampling is weighted by the square root of each label's positive count so rare labels are not oversampled into overfitting.

Splits are grouped by API endpoint, never by row, so near-duplicate documents from one endpoint cannot straddle training and evaluation.

How a document is scored

Text is cut into 512-token windows that overlap by 128 tokens, so each window advances 382 tokens. Every window is encoded, its token states are averaged under the attention mask, one linear head produces a logit per label, and the document takes the maximum over windows for each label. There is no cap on the window count, so input length is not limited.

512 is a measured choice, not a capacity limit. The encoder natively supports 8192 positions, but longer windows scored worse — and, against intuition, worst on the long documents they were supposed to help. Two poolings were tried to separate "dilution" from "length"; both lost, so it is the length. A single 8192-token pass was also 5.7x slower than the windowed path on an 8.3k-token document, because attention is quadratic.

Evaluation

Results, all 23 labels

Real traffic, 4,359 held-out documents, at the shipped threshold. Baseline AP is the same encoder fine-tuned without the domain-adaptive pretraining stage, on the same split, so the last column shows what that stage bought for each label:

Read the positive counts alongside the scores. Several labels rest on a handful of examples: a 1.000 on age means one positive was ranked correctly, not that the label is solved. The numbers are tight only where the count is large — product_data (1127), person_name (194), email (83), user_id (82), username (81), phone_number (72), credential (50).

Domain-adaptive pretraining helped most where labelled data was scarcest: user_id 0.326 → 0.962, payment_card_number 0.369 → 0.823, device_id 0.529 → 0.898, credential 0.565 → 0.901. It changed little where data was already plentiful (product_data, email).

Labels and thresholds

Label names are English identifiers. The training schema was authored in Chinese and config.json carries the mapping under label_names_zh, so the original names are recoverable: product_data ↔ product_data, credential ↔ credential, and so on. Renaming them changed only metadata — the head's 23 outputs keep their order and every score is bit-identical.

Speed

CPU, 8 threads, single process, one document at a time:

Overall median 339 ms, p90 16.8 s, p99 73 s. The distribution is heavily right-skewed — quote the median and the p90 together. Peak inference memory 3.4 GiB. Cap the windows per document if you need bounded latency.

Illustrative examples

The accuracy numbers above come from real traffic. The snippets below exist only to show the input and output shape, and to expose one failure mode — they are not evidence of accuracy, and a hand-written snippet is the worst case for this model. Every value in them is fabricated: example.com is reserved by RFC 2606, 203.0.113.0/24 by RFC 5737. The outputs shown are what the model actually returns for these strings.

A shipping-address response

Input, exactly as it comes off the wire:

{"code":0,"data":{"addresses":[{"address_id":"778901","receiver_name":"Andi Saputra","phone":"+62 812-3456-7890","detail":"Jl. Melati Raya No. 42, RT.003/RW.005","city":"Jakarta Pusat","province":"DKI Jakarta","post_code":"10510","is_default":true}]}}

Output: ["phone_number", "person_name", "street_address", "postal_code"], one window.

address_id does not trigger user_id: a record id is not a person id. province alone is an administrative division, which belongs to a label outside this schema, so street_address fires on the street line only.

Other shapes

A failure worth understanding

An order-detail response carrying a name, phone, address, e-mail, user id, product line, masked card number and client IP returns only ["phone_number","person_name","street_address","username","user_id"]. A human would also label e-mail, postal code, product data, IP and card number. All five are missed.

It is not that the values are unrecognisable. Each of them, alone, scores 1.000. Adding fields one at a time shows what actually happens:

Labels suppress each other. Nothing about the values changed; only the fields around them did. The likely mechanism is the training objective: each label back-propagates through its own single strongest window, and every label reads the same pooled vector for that window, so the high-frequency labels — person_name, phone_number, street_address, with hundreds to thousands of training positives — crowd out the rare ones. payment_card_number has 83 positives in the traffic portion of the training data and is the first to collapse.

Field naming, masking and document length are not the cause, which is worth stating because they are the usual suspects: payment_card_number scores 1.000 on 4111 **** **** 1234, on the unmasked number, and on a last-four-only form; ip_address scores 1.000 under client_ip, ip, remote_addr, X-Forwarded-For, as a bare value, and inside a sentence; email holds 1.000 with 300 unrelated records and 25 windows appended.

One miss has a different cause: product_data goes from 0.079 to 0.997 when 300 order-history records are added. The label was defined around a document being product data, not around a product name appearing once, so a single line item legitimately does not qualify.

Technical Specifications