meridianpii-te-v1

MeridianPII — Telugu PII Detection (v1)

On-device PII detection for Telugu (తెలుగు script), romanized/code-switched Telugu, and Indian-English text. Fine-tuned from microsoft/Multilingual-MiniLM-L12-H384, vocabulary-trimmed to a 54 MB INT8 ONNX that runs in the browser via transformers.js. A Telugu locale pack in the MeridianPII family — route Telugu text here; see meridianpii-hi-v2 for Hindi.

Extends Rampart's architecture to Telugu, a script Rampart's tokenizer cannot represent at all.

Evaluation

Span-level, on a frozen 1,500-row Telugu test set (Telugu-script · romanized · Indian-English), template-disjoint from training. Recall is the headline — a missed entity is a leak.

No general-purpose tool clears 62% on Telugu script; the purpose-built pack is +37 points over the best of them, at a fraction of the size and fully on-device.

Recommended pipeline

Ship alongside the deterministic recognizer layer (regex + checksums) that owns structured IDs — Aadhaar (Verhoeff), PAN, IFSC, GSTIN (mod-36), vehicle registration, voter ID, +91 phones, email, URL (national formats, shared across Indian locales). These are premasked in training so the model spends capacity on Telugu names/addresses/cities/contextual PII. Reference: github.com/plingampally/meridianpii. Keep-set CITY/STATE/ZIP are detected but retained.

Usage

import { pipeline } from '@huggingface/transformers';
const pipe = await pipeline('token-classification', 'plingampally/meridianpii-te-v1', { dtype: 'q8' });
const out = await pipe('నా పేరు వెంకట రావు, ఫోన్ 9876543210', { aggregation_strategy: 'simple' });

Use aggregation_strategy="simple". ONNX: onnx/model_quantized.onnx (INT8). Recommended confidence floor 0.15 (recall-biased).

Limitations

• Scope is Telugu-locale by design (locale pack); route other languages to their packs.
• CITY/STATE frequently interchanged (both keep-set → no redaction impact).
• Non-Indian geography out of scope. Structured IDs need the deterministic layer.
• Test set is synthetic; evaluate on your own data.

Labels (35 BIO = O + B-/I- × 17)

GIVEN_NAME, SURNAME, EMAIL, PHONE, URL, TAX_ID, BANK_ACCOUNT, ROUTING_NUMBER, GOVERNMENT_ID, PASSPORT, DRIVERS_LICENSE, BUILDING_NUMBER, STREET_NAME, SECONDARY_ADDRESS, CITY, STATE, ZIP_CODE (EMAIL/URL/TAX_ID/ROUTING premasked at serve time).

Training & license

Base microsoft/Multilingual-MiniLM-L12-H384 (Apache 2.0), vocab trimmed 250k→89k pieces (Latin + Telugu + digits, 100% output parity). Data: ~25k rows — Telugu-script / romanized-Telugu / Indian-English synthetic (template expansion over Telugu name/geo/format fillers). NFC normalization (never NFKD — it strips Telugu gunintams/matras). CC BY 4.0. Derives from Rampart (CC BY 4.0) — architecture, schema, premasking methodology.