hank-safe-harbor

<!-- Not a standard single-model `transformers` repo: a custom 2-encoder pipeline loaded via `SafeHarbor.from_pretrained` from the `hank-safe-harbor` package. Weights live under weights/ner-*, shipped as BOTH safetensors (torch) and ONNX (onnxruntime); there is no root config.json, so `AutoModelForTokenClassification.from_pretrained` / the widget do NOT apply. -->

hank-safe-harbor — HIPAA Safe Harbor PHI De-identifier + Consistent Faker (small · local · CPU or ≤24 GB GPU)

A small, fully-local, open clinical PHI tool that doesn't just blank out [REDACTED] — it rewrites every identifier into a consistent, type-faithful fake so the note stays readable, and it reports an honest end-to-end leak metric (the real value's verbatim survival in the final faked text, measured against ground truth — never a recall-path proxy). Detects competitively with the best open clinical encoders, runs on one ≤24 GB consumer GPU with zero PHI egress, and is the smallest open neural detect-and-consistently-fake pipeline we know of with a ground-truth leak metric.

⚠️ Read the scope honestly: all numbers below are measured on synthetic clinical notes (we have no licensed real-clinical corpus yet — see Evaluation scope). Recognized public-benchmark numbers (i2b2/n2c2) and a real-clinical result (PhysioNet) are in progress, not yet claimed.

---

TL;DR

• One system: an ensemble of two encoder NERs — roberta-large (English/Latin + diacritics) ∪ xlm-roberta-large (multilingual; non-Latin scripts) — feeding a deterministic coref + consistent-faker apply layer. (A generative path existed in development; it is not shipped in v1 — it scored below this ensemble and wasn't portable off Apple Silicon.)
• Recall ≥ 0.97 and per-value true-leak ≤ 0.03 on every (synthetic) bed, flat, no per-bed spike (all systems scored by one committed scorer on one denominator — non-Latin values included).
• Runs on one ≤ 24 GB consumer GPU (encoders ~1.8 GB at fp16; CPU-capable). Zero PHI egress.
• We do NOT claim raw-detection-F1 SOTA. A strong clinical encoder (Stanford) edges us on one bed; we own it. The contribution is the combination: local, open, detect and consistent faking, with an honest ground-truth leak metric.
• Evaluated on synthetic data only (so far). Public-benchmark (i2b2/n2c2) and real-clinical (PhysioNet) results are planned; see Evaluation scope & roadmap.

What "leak" means here (the honest metric)

Leak = a real ground-truth PHI value surviving verbatim in the final faked text, scored against the ground-truth values (not against the model's own detections — that would be tautological), and counted per distinct (type, value). This is what a downstream reader of the de-identified note would actually be exposed to. No non-shipping crutch (fail-closed gate, occurrence-propagation bridge, or a post-hoc recall-union outside the published pipeline) is added to flatter it; the structural recall-union described under How it works ships and is counted as part of the pipeline. The numbers are the raw shipped pipeline's output.

Headline benchmark (synthetic beds, one scorer for every system)

Every system below is scored by one committed, reproducible scorer (scripts/rebench_apples.py → benchmark/results/rebench_head_to_head.json) on one GT denominator per bed — normalize-only dedup with non-Latin (CJK/Greek/Thai) values included (an earlier internal harness used a gate that silently dropped them; this does not). Beds (note counts): in-dist 120 · fresh-OOD 36 · holdout-2 (intl) 30 · Nemotron 180. Small — treat single-bed deltas as indicative, not statistically significant (bootstrap CIs planned). All four are synthetic, including Nemotron (NVIDIA-generated synthetic PII), not real de-identified clinical records.

Our ensemble — recall + true end-to-end leak (verbatim GT-value survival in the faked text):

Detection recall vs open clinical encoders (same beds/scorer; obi & Stanford are detect-only, so their "leak" is the redact-all floor = 1 − recall, not a faked-text leak):

Stanford wins Nemotron — disclosed. We lead on the other three, and the largest gap is on holdout-2, the international bed: 0.990 vs 0.88/0.87 — and this is now an honest win: every system is scored on the same gt=940 denominator that includes the 65 non-Latin (CJK/Greek/Thai) values the English encoders miss and our XLM-R member catches. (n=30 — indicative, not significant.)

Fair-fight vs the real open clinical peer (riggsmed/deid-LONGFORMER-NemPII). Run through the same scorer/denominator as us (detect → consistent faker): we beat it 12–30× on end-to-end leak on the three beds outside its training distribution — and, critically, even on Nemotron, riggsmed's own training distribution, detection is a tie (ours 0.986 vs 0.988) and we still leak less (0.012 vs 0.015). Off its distribution riggsmed behaves like a general-PII detector (recall 0.70–0.79). Per-bed leak: ours 0.008/0.025/0.010/0.012 vs riggsmed 0.224/0.317/0.286/0.015. (benchmark/results/rebench_head_to_head.json.)

Context — vs a general-PII baseline (Microsoft Presidio + Faker). Not a clinical tool (clinical recall 0.72–0.93 here), so a weak-baseline reference rather than a peer (measured on the earlier scorer, not re-run on the unified denominator): end-to-end faked-text leak roughly 0.09–0.29 vs our 0.008–0.025 — an order of magnitude, but the riggsmed fair-fight above is the meaningful comparison.

Reproducible from scripts/rebench_apples.py → benchmark/results/rebench_head_to_head.json. (The earlier writeup docs/results/2026-06-18-phi-detection-benchmark-h2h.html predates this unified re-measure.)

Independent public benchmark — ASQ-PHI

ASQ-PHI (MIT) is a third-party benchmark of 1,051 synthetic clinical search queries (832 PHI-positive + 219 hard negatives) — a different distribution from our training notes, and it shares our policy that age < 90 is not PHI. We had never seen it.

*Comprehend = ASQ-PHI authors' own published baseline (validation_results/, threshold 0.3; cite-only, no API keys). obi / Stanford / ours re-run by us on the same beds. (ASQ is English-only — 0 of its 2,973 GT values are non-Latin — so its scorer and the unified scorer above share the same denominator.)

Honest read: obi and Stanford detect ~3 points more PHI than us on these short queries — but they do it by over-redacting 27–90 % of the zero-PHI hard negatives (flagging age/diagnoses), which destroys query utility — the exact "Safe Handoff" failure ASQ-PHI was built to expose. We sit at a far better point on that recall-utility tradeoff: recall on par with Comprehend (0.966 vs 0.968) while over-redacting 2.3 % vs ~90 %. Our raw detection recall is not SOTA on this set — we own that; our utility is. (benchmark/results/asq_phi_head_to_head.json.)

Coref + faking fidelity (LFF — the differentiator)

Measured on the actual faked output (in-dist, n=40): type-fidelity 0.985 (n=904 aligned spans); NAME/FACILITY consistency 0.955 (HOSPITAL 1.0 — one real facility → one fake across OCR/case variants; names 0.947); distinctness 0.884. Format-realism 1.0 but mostly tautological — only ~10 % of spans (EMAIL/IP/URL/ADDRESS) use independent validators; the rest are by-construction or weak (name = alpha-token only). Disclosed per-type in the JSON.

Over-redaction (false-positive faking) = 0.177 — about 1 in 6 faked spans is non-PHI text the recall-first design rewrote rather than risk a leak. This is a deliberate recall-over-precision tradeoff; it can rewrite some non-PHI content. Measured and reported; a tags-only (no-rewrite) mode and a utility-loss metric are planned.

How it works (the shipped pipeline)

1. Detect — union of two encoder NERs: roberta-large (English/Latin + diacritic-hardened) ∪ xlm-roberta-large (multilingual; closes non-Latin scripts). The union beats either encoder alone on every bed.
2. Structural recall-union — a set of high-precision syntactic detectors (EMAIL/URL/IP/SSN/ PHONE/FAX/DATE/MRN/ID-token formats) is unioned in to catch well-formed identifiers the model missed. This ships and is part of the reported recall; it is a syntactic recall-union, not a semantic crutch. A model-only vs model+structural ablation (to quantify each component's recall contribution) is planned.
3. Coref + apply layer — group mentions of one entity (deterministic, shared-significant-token) and replace each with a consistent, type-faithful, format-preserving fake (facility/name/date/ ID handlers). No [REDACTED] holes.

Deployment

• Footprint: two encoders (roberta-large 355 M + xlm-roberta-large 550 M) ≈ 1.8 GB at fp16 (~3.6 GB fp32 on disk). Comfortably within a single ≤ 24 GB consumer GPU; CPU-capable.
• Throughput (Apple-MLX dev box; a discrete GPU is faster): ~2,200–3,400 notes/hr; ~1.0–1.8 s/note.
• Cost anchor ≈ $35 / 1 M notes on a 4090 (re-derivable), vs cited cloud list prices (AWS ≈ $14.5k, Azure ≈ $13.1k / 1 M). Zero PHI egress.

Usage

Install the hank-safe-harbor engine plus a device-appropriate inference backend (so CPU users never pull CUDA):

pip install "hank-safe-harbor[onnx]"      # CPU, lightest (~40 MB rt) — Linux/Railway
pip install "hank-safe-harbor[onnx-gpu]"  # NVIDIA CUDA
pip install "hank-safe-harbor[torch]"     # Mac/MPS, or CUDA via torch
# (until on PyPI: pip install "hank-safe-harbor[onnx] @ git+https://github.com/hank-ai/hank-safe-harbor")

from hank_safe_harbor import SafeHarbor
deid = SafeHarbor.from_pretrained()                  # default model, downloaded on first use
# or an explicit repo id: SafeHarbor.from_pretrained("hank-ai/hank-safe-harbor")
out  = deid.deidentify(note_text)                    # -> {"deid_text", "spans", "mapping", ...}

# point at a local clone, force a backend, or list registered variants:
#   deid = SafeHarbor.from_pretrained("./hank-safe-harbor", backend="onnx")
#   from hank_safe_harbor import list_models; list_models()

The bundle ships each encoder as both model.safetensors (torch) and model.onnx (onnxruntime); from_pretrained auto-selects whichever backend you installed.

Limitations & out-of-scope (read this)

• Evaluated on synthetic data only (so far). Every number above is on synthetic notes (incl. Nemotron, which is NVIDIA-generated synthetic PII — not real de-identified records). For a clinical tool the audience cares most about real-note performance; we are pursuing PhysioNet (real-clinical) and i2b2/n2c2 (the recognized public benchmark) — see roadmap. Until those land, read the numbers as "strong on synthetic clinical text," not a real-world or SOTA claim.
• Not raw-detection-F1 SOTA. Strong English clinical encoders beat us on some beds; cloud systems may too. Our claim is the local + open + detect+fake + honest-leak combination.
• Prior art exists for detect+fake. Consistent-surrogate de-identification is not new (CliniDeID, PhysioNet De-ID, Stanford's hide-in-plain-sight). Our specific contribution is the smallest open neural pipeline doing it locally with a ground-truth leak metric — not "the only" tool that fakes.
• Over-redaction ~0.18 (false-positive faking) is the cost of recall-first; it rewrites some non-PHI. Measured and reported above.
• Leak is not zero. ~1–3 % per-value survival on these beds sits inside the cited Carrell HIPS 1–6 % band (context, not a superiority claim). Consistent faking is also vulnerable in principle to a Carrell-style "parrot" re-identification attack (attenuates, does not eliminate); an explicit parrot-attack audit is planned.
• Ages are NOT treated as PHI (including 90+) — a deliberate, documented deviation from HIPAA Safe Harbor for clinical utility; dates ARE redacted/faked. Disclosed up front.
• Small beds (tens to ~180 notes) are high-variance; bootstrap CIs are being added. Cloud competitors are cite-only (no API keys).

Evaluation scope & roadmap

• Now (v1): synthetic beds (above) + an apples-to-apples open-encoder head-to-head, all re-run through one scorer.
• Done: apples-to-apples fair-fight vs the riggsmed clinical detect+fake peer + the independent ASQ-PHI public benchmark (both above), all re-run through one scorer.
• In progress: TAB (recognized anonymization benchmark, off-domain cross-check); bootstrap CIs on every row.
• Planned (real-clinical): PhysioNet de-id gold corpus (real clinical text).
• v1.1: i2b2/n2c2 2014 strict-entity micro-F1 (official scorer), vs obi/Stanford, when the data portal reopens. We will report the full entity/token × full/HIPAA grid, not a single inflated number.

Provenance & licensing

Trained on synthetic clinical notes (teacher-distilled, schema-validated, decontaminated against all evaluation beds); no real patient data was used in training or evaluation. Base models (roberta-large, xlm-roberta-large) and their licenses are listed in NOTICE. Released under Apache-2.0 (see LICENSE). Eval harness + per-bed/per-type result JSONs in benchmark/results/; every metric here is an observed value from those files — none fabricated.