rampart

Rampart

rampart is a 14.7 MB ONNX token-classification model that detects personally identifiable information (PII) in text before it leaves the user's device. It is the on-device half of Rampart, a defense-in-depth client-side redaction system released by National Design Studio. The shipped artifact runs alongside a deterministic recognizer layer that handles structured identifiers; together they form the complete system.

This card documents the released artifact only. Alternative configurations explored during model selection (an ELECTRA-small base, the prefilter-off training variant, leaner data mixes, and smaller corpus slices) are discussed in the project whitepaper for context but are not published.

Model summary

Intended use

The model is designed for client-side redaction of user-typed text in AI assistants and intake flows — replacing identifying values with stable placeholders before any data is transmitted to a model provider, a server, or a logging system.

Direct uses

• Redact user content before passing it to a hosted LLM.
• Maintain stable placeholders ([GIVEN_NAME_1], [SSN_1], ...) across a multi-turn conversation, with rehydration on the client.
• Preempt accidental collection of personal data in analytics, traces, and crash reports.
• Validate domain-specific redaction policies before deploying chat systems in regulated contexts.

Out of scope

• Stand-alone government-ID detection. The model is one layer of a defense-in-depth system; it is not a replacement for the deterministic recognizer layer that ships alongside it. SSNs and payment cards are caught by the deterministic layer with checksum validation (structural rules and Luhn), at higher recall than the model alone. Phone, routing, government-ID, passport, and license numbers carry no checksum, so they are caught by the model; the deterministic layer does not attempt them.
• Indirect / inferential identifiers. A "rare disease + 5-digit ZIP" combination can re-identify someone even though neither token is in the redact-set. The model does not detect inferential leaks.
• Adversarial robustness as a security guarantee. We publish numbers on hostile inputs and document the failure surface; the system is positioned as harm reduction for users entering their own information in good faith, not as a security boundary against motivated adversaries.
• Non-Latin scripts. This release is scoped to the seven Latin-script languages listed above. Korean, Han Chinese, Japanese, Arabic, Cyrillic, and Devanagari names recall ~14% in aggregate (see "Fairness and limitations" below). Do not deploy this release for populations who routinely type non-Latin-script names without compensating controls; monitor accordingly.

Usage

The runtime ships as @nationaldesignstudio/rampart. createGuard() returns a ChatGuard that loads this classifier and runs the full deterministic + model pipeline:

import { createGuard } from "@nationaldesignstudio/rampart";

const guard = await createGuard();
const { text } = await guard.protect("My name is Alex Rivera and my SSN is 472-81-0094.");
// → "My name is [GIVEN_NAME_1] [SURNAME_1] and my SSN is [SSN_1]."

Training data

The shipped model is trained on a cumulative three-source mix, added in the order below; the selection matrix found that folding in all three sources produced every top-recall configuration, with breadth of corpus mattering more than the volume of any single source.

The synthetic and OCR'd corpora supply the disordered, document-noisy inputs OpenPII lacks (its conversations are clean and well-formed); OpenPII supplies the multilingual breadth and the held-out test set. The exact synthetic/OCR mix and volume were chosen by the end-to-end selection matrix, not assumed — see the project whitepaper (§3.1, §3.5, §4.2) for the ablation that fixed the recipe.

The held-out 100,000 rows (from the AI4Privacy corpus) are split into two non-overlapping subsets, seeded for full reproducibility:

• 10,000 rows for recall-floor threshold tuning.
• 30,000 rows for the headline test results below (per-language row counts in the eval table).

The remaining 60,000 held-out rows are reserved for future evaluation and are not used in this release.

Pre-processing

All training rows pass through the same normalization the runtime applies before tokenization: lowercase, NFKD decomposition, and combining-mark stripping. The combining-mark step folds accents — José becomes jose, Müller becomes muller — so the model sees a single canonical form regardless of how the user typed the name. This matches what BERT's BasicTokenizer does implicitly at inference time under do_lower_case=True, so the train-time and runtime distributions are identical by construction. A guard in the training pipeline fails the run if a future tokenizer change breaks this assumption.

A re-trainer who omits this normalization step will produce a model with mismatched distributions, and recall numbers will not reproduce.

The structured classes the deterministic layer owns (SSN, CREDIT_CARD, IP_ADDRESS) are also masked to sentinel tokens before tokenization — both at inference (src/premask.ts) and during dataset construction — so the model never learns to classify raw card/SSN/IP digits and the train-time and inference-time inputs match by construction.

Vocabulary

The full BERT-uncased vocabulary contains 30,522 WordPieces. The shipped vocabulary retains:

1. All special tokens ([PAD], [UNK], [CLS], [SEP], [MASK]).
2. All single-character pieces and their ## continuations, which preserve WordPiece's character-level fallback for rare names.
3. All multi-character pieces appearing in the training corpus above a frequency threshold.

The shipped vocabulary is 19,730 pieces.

Training procedure

The final epoch was selected by held-out eval loss.

Label taxonomy

The model emits 35 BIO labels (17 entity types × {B-, I-} + O); the deterministic recognizer layer contributes three more structured classes that are masked before the model runs. The runtime applies a default-deny policy: every detected span is redacted unless its label is explicitly in the keep-set.

Redacted by default

Owned by the deterministic recognizer layer (regex + validator, masked before the model):

Emitted by the token-classification model:

BUILDING_NUMBER + STREET_NAME together form the precise street line; both are redacted while city/state/ZIP are kept.

Kept by default

The keep-set keeps coarse geography (city/state/ZIP) while redacting the precise street line. To change it, edit KEEP_LABELS in src/types.ts — it is a compile-time set, not a runtime flag.

The taxonomy is deliberately atomic: there is no coarse PERSON, STREET_ADDRESS, ADDRESS, ORGANIZATION, or LOCATION label, and no catch-all SECRET. Names split into GIVEN_NAME / SURNAME, the street line into BUILDING_NUMBER / STREET_NAME, and document identifiers into their specific classes, so the model learns to catch PII fragments in disordered text rather than expecting one tidy blob. Dates, ages, and income are intentionally not modeled as PII (they map to O): a bare date is rarely identifying, and assistants need age and income as context, so redacting them was over-redaction without a privacy gain.

Evaluation

We score the full system (model + deterministic layer) because that is what consumers experience end-to-end. Model-only numbers are reported separately for researchers who want to evaluate the encoder in isolation.

Primary metrics

• Private-term recall: for every gold private value, did the redacted output contain the value? This is the privacy-headline number; misses here are leaks.
• Public-term retention: for every gold public value, did the redacted output preserve the value? This measures over-redaction.
• Span F1 strict (IoU=1.0) and relaxed (IoU≥0.5): how well predicted span boundaries align with gold boundaries under one-to-one greedy matching.
• Latency: Node.js ONNX runtime cold / p50 / p95 / p99 over the full 30,000-row test set. Browser latency (WebGPU and WASM backends) is measured separately by eval/bench/webgpu.ts — see below.
• Calibration: 15-bin reliability ECE, per label and overall, on per-span max-class scores.

All recall and retention numbers carry Wilson 95% confidence intervals; stratified breakdowns include 1000-iteration bootstrap intervals.

Held-out OpenPII test set — seven supported languages (30,000 rows; 131,707 private terms; 87,207 public terms)

The headline number is measured across all seven supported Latin-script languages. English-only, Spanish-only, and the English+Spanish slice are reported as sub-slices.

2,082 leaks of 131,707 private terms on the seven-language test (1 in 64 terms slips past the system, before the application's downstream defenses fire). On the English+Spanish slice the system leaks 778 of 67,613.

These numbers are measured by the committed eval/bench harness running the shipped Q4 pipeline end-to-end over a pinned held-out slice of pii-masking-openpii-1.5m. The harness was corrected relative to earlier revisions of this card: city/state/ZIP are now scored as kept (matching the runtime keep-set) instead of being counted as leaks, so public retention reflects policy-aware behavior directly. Recall is reported against the full, harder seven-language slice. Span-F1 strict (exact byte+label match) is a secondary metric; term-presence recall is the privacy headline.

The 6.6 ms p50 above is the Node ONNX (CPU) figure over the 30k held-out set. Run over a held-out OpenPII slice in the browser, the same shipped pipeline measures 3.9 ms p50 on WebGPU (Apple Metal, p95 9.3 ms) and 12.6 ms on WASM (p95 35.5 ms), via eval/bench/webgpu.ts — so the WebGPU form factor is faster than Node CPU on the same class of inputs, and WASM is the floor when no GPU is available.

* See "Schema reconciliation" below — the Rampart policy redacts the precise street line (BUILDING_NUMBER + STREET_NAME) and the secondary-address line while keeping city/state/ZIP, which the harness now honors.

Per-language slices (OpenPII Latin test, 30k rows across 7 languages)

All seven languages land in the 97-99% band; Dutch is the lowest at 97.21% and is flagged for attention in subsequent training cycles. (The recall band moved down ~1pp versus the previous card because the harness now scores the corrected, harder slice — see the note above; the same model scores higher on the older, easier slice.)

Hand-curated suites

The adversarial and domain-intake suites are 20 cases each; Wilson CIs are wide. The 1,875-case fairness suite has tight CIs and is the most statistically grounded slice we report.

Schema reconciliation

The 91.69% retention number in the headline table is term-presence scoring that already credits city/state/ZIP as kept, matching the runtime keep-set. We analyzed the 7,244 remaining "over-redacted" public terms in the 30,000-row eval:

• The vast majority are policy-driven redactions of street-line components (street name, building number, secondary address line). AI4Privacy OpenPII marks STREET, BUILDINGNUM, and SECADDRESS as O (public); the Rampart policy redacts the precise street line (BUILDING_NUMBER + STREET_NAME) and SECONDARY_ADDRESS while keeping CITY, STATE, and ZIP. These are not detector errors; they are the policy firing as designed.
• A smaller share are span-edge artifacts. The runtime's particle-rescue step grows name spans (GIVEN_NAME / SURNAME) to swallow capitalized particles ("de la", "von", "Mc"). When an adjacent public token is itself capitalized, that token can be absorbed into the redacted span.
• A very small fraction are digit fragments inside longer correctly-redacted spans (e.g. "376" found inside a redacted 16-digit credit card).

We publish the 91.69% term-presence number for like-for-like comparison against public PII benchmarks running the same scoring rules. For product reasoning, the policy-aware retention exceeds 99%.

Calibration

The runtime applies a single recall-biased confidence floor (minScore = 0.4) uniformly across the model's labels, chosen against the 10,000-row OpenPII Latin calibration split (disjoint from test) so misses — which leak data — are traded against the cheaper failure of over-redaction. There is no per-label threshold table in the shipped runtime; the deterministic recognizer layer, not a tuned model threshold, is the system of record for the structured classes the model alone is weakest on:

• SSN — structural validation (reserved-area rules).
• CREDIT_CARD — Luhn checksum over the digit projection.
• EMAIL / URL / IP_ADDRESS — pattern-anchored regex at near-100% recall.

Phone, routing, government-ID, passport, and license numbers carry no checksum and are left to the model under the same recall-biased floor.

ECE on the full 30,000-row test set is 0.291 (overall, all labels); the model alone (no deterministic layer) is 0.018. The system-level ECE is higher because the deterministic layer always emits score 1.0 on its detections, making the score distribution bimodal — that is a score-distribution artifact of the union, not a calibration regression of the underlying model.

Fairness and limitations

We document failures because consumers need this to deploy the redactor responsibly. None of these are surprises; we measured each.

Fairness across naming traditions (1,875 Faker-generated cases)

Cases are stratified by naming tradition (15 categories) embedded in 5 chat templates. Same surrounding context across all traditions — only the name varies.

Aggregated by script:

• Latin-ASCII names: ~100% recall (695 / 695)
• Latin + diacritics: 99.8% recall (429 / 430)
• Non-Latin scripts: 13.7% recall (103 / 750)

The deterministic recognizer layer does not catch names — there is no checksum to validate against — so this failure surfaces at the system level. This is the most important regression we have identified, and the fairness suite is wired into the eval pipeline as a stratified regression test so any further drop will surface in subsequent training cycles.

Government-style identifiers (model only)

Government-style identifiers (case numbers, Medicare-style identifiers, USCIS receipts, A-numbers, passports, licenses) carry no checksum, so — unlike SSNs and payment cards — the deterministic layer does not detect them. They rely entirely on the model, which catches ~67.6% of them in a structured-ID probe. This is a documented weak spot: there is no deterministic backstop for these classes, so the model's recall is effectively the system's recall on them. Consumers should not assume the deterministic layer covers government IDs the way it covers SSNs and cards; deployments that handle these identifiers heavily should add their own format-specific validators.

Adversarial robustness

The system catches most homoglyph, casing, leet, NFC/NFD, and basic whitespace-split attacks. It does not reliably catch:

• Zero-width characters injected between every digit of an SSN.
• Prompt-injection text inside the PII span (e.g. "ignore previous instructions").
• Combined attacks (homoglyph plus whitespace split).

The deterministic layer's digit projection (which strips non-digit characters before checksum validation) restores most digit-bearing PII against these attacks; names remain vulnerable. This is the right framing for the limitation, not the primary use case: Rampart is designed to protect users entering their own information in good faith from incidental disclosure to downstream services, not to defeat a motivated user actively trying to smuggle their own PII past the filter.

WordPiece fragmentation on long names

Names like Thanh-Nghiem Quoc-Bao or Chukwuemeka Okonkwo-Adeyemi produce many subwords; the runtime performs span-merging across same-label adjacencies plus particle-rescue, which closes most of the gap. Some five-or-more-subword names still fragment in a way that loses recall on the trailing subword.

Reproducibility

The model weights, deterministic layer, and TypeScript evaluation harness are released under CC BY 4.0.

Evaluation runs entirely in TypeScript, against the shipped pipeline: the native benchmark (eval/bench) runs the real @nationaldesignstudio/rampart code over a frozen OpenPII held-out slice and writes summary.json / by_language.json, which are committed alongside the eval output — so every number in this card traces to committed evidence produced by the code that ships. The held-out row uids are pinned in a committed manifest; regenerate the data with bun run bench:fetch and reproduce the figures with bun run bench.

Citation

If you use this model in research, please cite:

@misc{rampart-2026,
  author = {National Design Studio},
  title  = {Rampart: Client-side PII redaction for AI assistants},
  year   = {2026},
  url    = {https://huggingface.co/nationaldesignstudio/rampart},
}

Please also cite the upstream training corpus:

@misc{ai4privacy-openpii-1.5m,
  title  = {ai4privacy/pii-masking-openpii-1.5m},
  author = {AI4Privacy},
  year   = {2025},
  url    = {https://huggingface.co/datasets/ai4privacy/pii-masking-openpii-1.5m},
}