koelectra-ko-pii-ner

koelectra-ko-pii-ner · Korean PII Detection Model

A 14M-parameter NER model that detects personal data in Korean text and returns its position and type.

Used for masking personal data, or for identifying it before text is sent to an external LLM or to the cloud.

none does not mean the sentence contains no personal data. It means an entity that is detected but not treated as personal data, such as PLACE or ORGANIZATION.

Usage

import torch
from transformers import AutoModelForTokenClassification, AutoTokenizer

MODEL_ID = "atonlee/koelectra-ko-pii-ner"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForTokenClassification.from_pretrained(MODEL_ID).eval()

text = "김민준 씨한테 010-1234-5678로 연락해줘"

inputs = tokenizer(
    text,
    return_offsets_mapping=True,
    return_tensors="pt",
    truncation=True,
    max_length=512,
)

offsets = inputs.pop("offset_mapping")[0]

with torch.no_grad():
    logits = model(**inputs).logits

predictions = logits.argmax(dim=-1)[0]

for pred, (start, end) in zip(predictions, offsets):
    if start == end:
        continue

    label = model.config.id2label[pred.item()]

    if label != "O":
        print(label, repr(text[start:end]), f"[{start}:{end}]")

Output:

B-NAME  '김민'  [0:2]
I-NAME  '준'    [2:3]
B-PHONE '010'   [8:11]
I-PHONE '-'     [11:12]
I-PHONE '123'   [12:15]
I-PHONE '4'     [15:16]
I-PHONE '-'     [16:17]
I-PHONE '56'    [17:19]
I-PHONE '78'    [19:21]

The model predicts personal data using the BIO scheme.

• B- — start of a region
• I- — continuation of the same region
• O — token that is not a target

Tokens are subword units, so consecutive BIO tags have to be joined to recover the actual region.

김민준                          NAME
010-1234-5678                   PHONE
900101-1234567                  RRN
5310-99**-****-1122             CARD_NUMBER
서울시 강남구 테헤란로 152 3층      ADDRESS

One joined region is one entity. 김민준 = NAME (0, 3) is one. A sentence can contain several, and the evaluation below counts entities, not sentences.

Labels

29 labels.

tier1 — do not send off-device

Identifies a person on its own.

tier2 — mask, then send

Identifies a person in combination with other data.

none — send as-is

Detected, but not treated as personal data.

ADDRESS and PLACE are split by form. With a building or unit number it is ADDRESS (tier2); without one it is PLACE (none).

ONNX

ONNX models are included.

model_quantized.onnx is the uint8 model under the name Transformers.js looks for.

Training

Training used public datasets and hand-written PII examples.

The following were written for this model and added to the training data.

• masked identifiers
• partially revealed identifiers
• ordinary sentences containing no personal data
• strings shaped like identifiers but not personal data

Neither the source datasets nor the merged training corpus is redistributed here.

Evaluation

5,171 test sentences carry ground-truth annotations — 17,537 entities in total, personal data included. Predictions were compared against those annotations.

Three criteria, differing in how much has to match.

By tier, on the type criterion.

Licence

Model weights: Apache-2.0

Inherited from the base model. Each external dataset used in training keeps its own licence.

• BCCard/pii-masking-openpii-finance — CC BY 4.0
• townboy/korean-pii-dataset — CC BY 4.0

This repository does not relicense or redistribute the source datasets.