ko-pii-detector-ax-crf

ko-pii-detector-ax-crf

Korean PII (Personally Identifiable Information) span detector — CRF tagging head on top of skt/A.X-Encoder-base. Requires trust_remote_code=True (custom ModernBertCRFTagger architecture) and pyt-crf package.

• Github: id4thomas/ko-pii-detector
• pyt-crf package: id4thomas/pyt-crf

pip install pyt-crf

Performance

Entity-level (exact span match) on held-out test set (14,294 examples):

Per label (f1, entity-level exact match):

DRIVER_LICENSE/GENERIC_ID are the weakest labels (high format variance, no fixed pattern to key on). This CRF head consistently outperforms the token-classification variant (id4thomas/ko-pii-detector-ax-tokenclf) across labels, likely from the transition constraints stabilizing span boundaries.

Full precision/recall per label, head-to-head comparison: ko-pii-detector docs/performance.md

Dataset

Trained on BCCard/pii-masking-openpii-finance with additional noise injection (typos, spacing/format variants, obfuscation, partial-value references, distractor IDs, etc. — see docs/noise.md) to make the model robust to real-world noisy input.

18 span labels:

Usage Example

import torch
from transformers import AutoModel, AutoTokenizer

# Load tokenizer, model (custom architecture -> trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("id4thomas/ko-pii-detector-ax-crf", use_fast=True, trust_remote_code=True)
model = AutoModel.from_pretrained("id4thomas/ko-pii-detector-ax-crf", trust_remote_code=True)
model.eval()

id2label = model.config.id2label

# predict
text = "문의사항은 010-1234-5678 또는 [email protected] 으로 연락 주세요."
enc = tokenizer(text, return_tensors="pt", return_offsets_mapping=True)
offsets = enc.pop("offset_mapping")[0].tolist()

with torch.no_grad():
    pred_ids = model(**enc, decode=True).predictions[0].tolist()  # CRF-decoded tag ids

# merge BIO-tagged tokens into entity spans
entities, cur = [], None
for pred_id, (start, end) in zip(pred_ids, offsets):
    if start == end:
        continue
    tag = id2label.get(pred_id, "O")
    if tag == "O":
        cur = None
        continue
    prefix, _, label = tag.partition("-")
    if prefix == "B" or cur is None or cur["label"] != label:
        cur = {"label": label, "start": start, "end": end}
        entities.append(cur)
    else:
        cur["end"] = end

for ent in entities:
    ent["value"] = text[ent["start"]:ent["end"]]
    print(ent)
# {'label': 'PHONE', 'start': 6, 'end': 19, 'value': '010-1234-5678'}
# {'label': 'EMAIL', 'start': 23, 'end': 39, 'value': '[email protected]'}

More Examples (per label)

One example per label from dataset's held-out validataion