schift-ko-pii-v6

schift-ko-pii-v6

34.1M parameter Korean PII detector built on a dual-path LoRA encoder.

The 0.4.0 release keeps the v6 detector contract and adds an optional, typed selective-adoption flow for structured and contextual categories. The base install does not install ko-pii; install the extended extra only when those additional deterministic categories are needed.

Release status

This source tree prepares schift-ko-pii 0.4.0. The previously published baseline was 0.3.3. The Cloud Run ONNX service under services/pii is a separate deployment lane; this package does not bundle its ONNX artifacts.

Quick start

pip install schift-ko-pii

The original detector API remains available:

from schift_ko_pii import detect

spans = detect("피고 김민수의 전화번호는 010-1234-5678이다.")
# [
#   {"start": 3, "end": 6, "label": "private_person", ...},
#   {"start": 14, "end": 27, "label": "private_phone", ...},
# ]

For a typed operational result, use the selective-adoption flow:

from schift_ko_pii import AnalysisConfig, ProcessingMode, analyze_text

result = analyze_text(
    "피고 김민수의 전화번호는 010-1234-5678이다.",
    config=AnalysisConfig(mode=ProcessingMode.PERMISSIVE),
)

# The result contains typed detections, policy actions, BLOCK-only masking,
# counts, and a metadata-only review queue. It has no separate source-text
# field; its policy text can still preserve REVIEW/ALLOW spans for operators.
print(result.summary)
print(result.masking.masked_text)
print(result.review_items())

Selective-adoption flow

The public workflow is deliberately ordered:

detect -> assess -> BLOCK-only masking -> review queue

analyze_text() or analyze() runs detection, sends non-sensitive detection metadata to assess(), and masks only spans whose action is Action.BLOCK. Action.REVIEW detections remain unmasked and are represented by ReviewItem values in PiiResult.review_queue; callers can create a typed FeedbackPatch with propose_feedback_patch() without persisting raw PII. Action.ALLOW detections are retained in the typed result but are not masked. The resulting masking.masked_text is therefore a policy output, not a safe untrusted-egress string: use a caller-owned all-span redaction step before sending it to logs, external APIs, or other untrusted surfaces.

Detector confidence and operational risk are separate. ProcessingMode sets the action thresholds:

PERMISSIVE is an action-policy choice, not a change to the v6 detector's score_threshold. Use AnalysisConfig.score_threshold separately when configuring detection.

Extended profiles (opt-in)

pip install "schift-ko-pii[extended]"

The optional adapter is enabled per request:

from schift_ko_pii import AnalysisConfig, analyze_text

result = analyze_text(
    "사업자등록번호 104-81-49532, 직책 팀장",
    config=AnalysisConfig(extended=True, extended_profile="contextual"),
)

extended_profile="structured" adopts deterministic identifier and anchor categories such as business/corporate registration numbers, medical insurance and prescription identifiers, PNU, postal code, fax, employee, document, petition, and drug IDs. extended_profile="contextual" includes that structured set plus contextual attributes such as nationality, birth date, education, major, position, age, height, and weight.

The taxonomy registry is exposed through LABELS, STRUCTURED_UPSTREAM_LABELS, CONTEXTUAL_UPSTREAM_LABELS, EXCLUDED_UPSTREAM_LABELS, lookup_label(), label_for_upstream(), and upstream_labels_for_profile(). The adapter excludes categories already owned by the v6 detector or current Schift postprocessing, including person, address, phone, email, existing structured identifiers, URLs, IPs, and legal case references. Existing v6 spans win overlaps, except a generic account_number span may be refined by a more specific extended label.

Masking boundaries

Masking is request-local and occurs only after policy assessment. Select a MaskingStrategy in AnalysisConfig or call mask_text() directly with typed MaskSpan values:

• TOKEN: replace with a stable label token such as [PII_PHONE_1].
• REDACT: replace with [REDACTED].
• PARTIAL: retain a small leading/trailing portion for recognition.
• HASHED: replace with a deterministic local SHA-256 digest.

These strategies are output transformations, not custody. No strategy stores a reverse map, restores source values, or talks to Vault. Central Vault custody, retention, tenant isolation, KMS, and audit requirements remain a separate caller-side project.

Documents and the document-helper boundary

Document APIs accept already extracted text and provenance, not files:

from schift_ko_pii import (
    ExtractedPageInput,
    analyze,
    from_pages,
)

document = from_pages(
    (
        ExtractedPageInput(page_num=1, text="첫 페이지", source="helper"),
        ExtractedPageInput(page_num=2, text="둘째 페이지", source="helper"),
    ),
    source_id="doc-123",
)
result = analyze(document)

from_text(), from_pages(), and from_document_helper() build the typed text-only envelope. DocumentInput preserves the concatenated text and page boundaries; SourceSpan and PageSpan preserve character offsets and page provenance. scan_document() is the convenience scan over that envelope.

The package does not parse HWP/HWPX, DOCX, XLSX, PDF, or other file formats. Use the document-helper service (or another caller-owned parser) to produce a text-only envelope, then pass it to this package. Do not treat the envelope as file storage or a Vault integration.

Postprocessing and legacy API

Postprocessing is enabled by default for detect(). It applies Korean-specific structured-ID validation, checksum checks where applicable, context-aware span merging, and false-positive suppression for legal case numbers and statute references. The legacy detect() path preserves original input text by default; pass normalize=True when you want NFKC-normalized model input and source-offset remapping. The typed analyze()/analyze_text() flow enables that normalization by default. Pass postprocess=False for encoder heads only (person, address, and organization).

Existing root exports remain available: detect, mask, apply, assess, detect_extended, detect_extended_entities, Action, ProcessingMode, RiskLevel, and ExtendedDependencyError.

For compatibility, AnonymizationResult is an alias of PiiResult, and both anonymize_text and anonymize are aliases of analyze_text. They do not introduce a second execution path.

API (free)

For production use without managing model files:

from schift import Schift

client = Schift(api_key="...")  # free at schift.io
result = client.pii.redact("김민수의 전화번호는 010-1234-5678입니다.")

Labels

The stable local taxonomy is available as immutable TaxonomyEntry values in LABELS. Common labels include:

Benchmark

The current V6 benchmark is benchmark/benchmark_v3.jsonl. The V6 release measurement on the current benchmark is:

Run the current benchmark with postprocessing enabled:

python benchmark/run_benchmark.py \
  --benchmark benchmark/benchmark_v3.jsonl \
  --postprocess

This is the current V6 benchmark surface. Historical benchmark datasets and selective-adapter policy matrices are not presented as additional model-card benchmarks here.

Model details

• Checkpoint: schift-io/schift-ko-pii-v6
• Architecture: dual-path LoRA encoder with person, address, and organization heads
• Training: LoRA adapter on Korean legal/financial/admin examples
• Format: safetensors release source
• Inference: custom transformers/PyTorch dual-path loader
• Max length: 512 tokens
• Tagging scheme: O/B/I/E/S

License

Schift License v2.0 — Apache 2.0 base with a revenue threshold. Free for everyone under $10M annual revenue. Research, education, and non-profit use always permitted. Companies above the threshold: contact [email protected].

Citation

@software{schift_ko_pii_2026,
  author = {Schift Inc.},
  title = {schift-ko-pii: Korean PII Detection Model},
  year = {2026},
  url = {https://huggingface.co/schift-io/schift-ko-pii-v6},
}