hushmark-modernbert-tr-149m

hushmark-tr-v2

Status

hushmark-tr-v2 is a closed-set multi-label BIO tagger for the twelve NER-owned Hushmark entity types. It passed the locked adoption gate on 2026-08-22 against the incumbent hushmark-tr GLiNER model. It is not yet the runtime default: the incumbent remains configured while the generalisation limits below are addressed.

Intended use

The model proposes spans for the twelve NER-owned types: person, address, organization, date of birth, health, religion, ethnicity, political opinion, sexual life, criminal record, biometric reference, and union membership. Deterministic identifiers and secrets remain owned by the L0 validators and are rejected at the corpus schema boundary, so the model is never trained to duplicate them.

This model is a detection aid, not an anonymisation or legal-compliance guarantee. False negatives and false positives remain possible; operators must validate it on their own data.

Special categories are subject-conditioned

Following the decision recorded in docs/hushmark-tr-v2-plan.tr.md section 3.1, the eight special categories are labelled only when attributed to a data subject. "Hipertansiyon Türkiye genelinde yaygındır" carries no data subject and is a negative; "Hasta, migren nedeniyle rapor talep etti" is a positive. The taxonomy descriptions in taxonomy/taxonomy.yaml were updated to match.

Architecture

ytu-ce-cosmos/modernbert-tr-base (Apache-2.0) with one three-way O/B/I head per entity type. Per-type heads rather than a single tag sequence: an ADDRESS containing an ORG is common in Turkish administrative text and a single sequence cannot represent it.

148,765,476 parameters. Context 384 tokens at training time; the architecture supports 8,192.

GLiNER was dropped because its value is open-vocabulary zero-shot recognition and this taxonomy is closed and pinned. The measured cost of that mismatch is in the comparison below.

Data provenance

Corpus SHA-256 93098439f539218b37cbd752441173141f2e4ace2d16e2d9690c97431318e3a0, 114,983 documents: train 51,275 · dev 4,421 · locked eval 5,513 · quarantine 53,774.

Span counts: PERSON 17,845 · ORG 9,687 · DOB 5,319 · ADDRESS 3,487 · HEALTH 1,799 · CRIMINAL 1,797 · UNION 1,797 · ETHNICITY 1,796 · RELIGION 1,795 · POLITICAL 1,788 · SEXUAL_LIFE 1,788 · BIOMETRIC_REF 1,790. Negative documents: 28,576.

Special-category surface forms come from a 550-entry lexicon (digest 343a2538cb20395c) rather than the three-per-type bank used for v1.

Split membership is derived from the upstream leakage group where one exists and from normalised content otherwise. Leakage audit on the trained splits: 0 exact collisions, 4 near duplicates.

Training

One NVIDIA A100-SXM4-80GB, bf16, batch size 32, max length 384, 3 epochs, encoder LR 3e-5, head LR 1e-4, 200 warm-up steps, outside-class weight 1.0 against 8.0 for B/I, validation every 300 steps, patience 6. The run took 575.5 seconds over 4,500 steps and peaked at 9,965,534,720 bytes of GPU memory. Run fingerprint a05350320ef27f927b19f6a0a643d99ebed882f7534e80e5251611198a3c9246.

Development-best macro strict-F1 was 0.9994392104 at step 2,700, and that checkpoint was selected.

Locked evaluation

The 5,513-row locked split was scored exactly once, after development selection. The evaluation script refuses to overwrite an existing verdict.

Absolute improvement +0.6293720926; minimum required +0.05; per-type regressions over 0.02: none; machine verdict adopt: true.

Per-type strict F1, candidate against incumbent:

The number that matters more

The locked split is drawn from the same generators as training, so 0.9961 measures whether the model learned the corpus — not whether it understands Turkish. A ten-item probe set written by hand outside the generator's sentence frames (bench/data/probes-tr-v1.jsonl) scores 8/10:

• in-frame 1/1, curated negatives 3/3, out-of-frame 4/6.
• "Hasta Ayşe Yılmaz migren tanısıyla başvurdu." — HEALTH missed. The adjectival ... tanısıyla construction does not appear in any training frame. A false negative on an attributed health condition is the most expensive error this model can make.
• "Doktor bey, iki yıldır tip 2 diyabet hastasıyım." — HEALTH found, but a spurious PERSON on the form of address.

Read the two numbers together: the corpus is learnable and the architecture works, and frame diversity — not lexicon size — is now the binding constraint on the eight special categories.

External anchors

Scored against four suites outside this model's generators, five other engines alongside it. Every engine saw identical text; the annotation conventions are not ours.

Read those four rows together and they say one thing. On the trap/twin pair — the same confusable surfaces in both directions — v2 is the strongest engine measured: it emits nothing on 342 subject-free look-alikes while recovering 0.8063 of the attributed twins. Nobody else comes close (gliner_multi_pii-v1 reaches 0.6873 recall but pays a 0.430 false-positive rate). The subject-conditioning described above is real and it is measurable.

On human-annotated encyclopedic prose it collapses to 0.2967, below the incumbent hushmark-tr at 0.4437. A separate probe over 212 human-annotated group mentions, split by whether a data subject is present, puts the discrimination ratio at 1.02 — indifference. The subject-conditioning does not transfer out of the administrative register.

So: use this model on forms, records and administrative correspondence, which is what it was built for and where it is genuinely good. Do not use it on free prose, news or encyclopedic text. The 0.9961 locked figure describes neither case; it describes the training distribution.

ONNX deployment

Opset-19 FP32 graph, 595,684,002 bytes, SHA-256 1be3d53cc51cf93b744331522608d5cb1b0fb5f62ecb2d8b66c05290a4eb13a4. The serving path needs no transformers and no torch: a pinned tokenizer.json drives the Rust tokenizer and onnxruntime drives the graph. Label order is pinned in labels.json and checked against the taxonomy order at load; a re-ordered checkpoint cannot be served silently.

On a stratified development sample covering all twelve types at threshold 0.5, FP32 scored macro strict-F1 1.0000 and dynamic INT8 scored 0.9998021 — a loss of 0.0002, far inside the 0.02 limit.

INT8 is viable for this architecture. v1's INT8 collapsed to 0.4136 on mDeBERTa-v3's disentangled attention; ModernBERT has no such structure. The INT8 graph is 149,825,638 bytes, SHA-256 f3849966184b43b1b5bd8dec6ac8b4d9fdb331fd87be12a77e31ec19ec6ffa6f. FP32 is registered as the pinned export; INT8 is a validated candidate the operator may switch to.

Latency

Local Apple ARM64 host, 4 torch threads to match the core container's 4 vCPU quota.

Artifact size drops from 1,157 MB (v1 FP32) to 596 MB (v2 FP32) or 150 MB (v2 INT8) — 87% smaller at INT8. Most of v1's weight was a 250,105-token multilingual embedding matrix, 66% of its parameters, replaced here by a 50K Turkish vocabulary.

This is technical performance evidence for one host profile, not a capacity guarantee.

Limitations

• Frame overfitting is the dominant limitation. Special-category positives are generated from a bounded set of sentence frames. Section "The number that matters more" quantifies the gap.
• The model is a register specialist, not a Turkish PII model. See "External anchors" above; this is the limitation that matters most when deciding whether to deploy it.
• The locked split shares generators with training. It is a regression gate, not a measure of production capability.
• The internal review round-trip (corpus/review.py) is implemented but has not been run, so no in-house human-adjudicated set exists. The external anchors below are human-annotated by third parties, which is not the same thing as agreement with our own annotation guideline.
• 53,774 real Turkish document chunks remain in quarantine, unannotated.
• ORG supervision is thin lexically: roughly 150 distinct organisation names, all public institutions. Private-company names are absent.
• Inputs beyond the configured context are truncated. Long-document chunking is unmeasured.
• The mined negatives are labelled for the eight special categories only; their organisation and address spans are unannotated and masked out of the loss.

License and release

The base encoder ytu-ce-cosmos/modernbert-tr-base is Apache-2.0. Artifact digests are pinned in core/models.yaml in the hushmark repository under hushmark-tr-v2. Weights are distributed separately from the public repository and must match the pinned hashes before use.

Files

Use

The serving path needs no transformers and no torch: a pinned tokenizer.json drives the Rust tokenizer and onnxruntime drives the graph. The head is one three-way O/B/I decision per entity type, so logits reshape to [tokens, 12, 3] in the order given by labels.json.

import json, numpy as np, onnxruntime
from tokenizers import Tokenizer

labels = json.load(open("labels.json"))
tok = Tokenizer.from_file("tokenizer.json")
session = onnxruntime.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])

encoding = tok.encode("Hasta Ayşe Yılmaz migren tanısıyla başvurdu.")
logits = session.run(None, {
    "input_ids": np.asarray([encoding.ids], dtype=np.int64),
    "attention_mask": np.asarray([encoding.attention_mask], dtype=np.int64),
})[0]
scores = logits[0].reshape(len(encoding.ids), len(labels), 3)
# tag 1 = begin, 2 = inside, 0 = outside; character offsets come from encoding.offsets.

Reference decoding is hushmark_core.ner.tokencls_decode; the runtime adapter that verifies size, SHA-256, and label order before loading is hushmark_core.ner.tokencls_backend.

Reading the evaluation folder

Start with evaluation/model-comparison.md. The locked verdict in evaluation/locked-eval-verdict.json says adopt: true at macro strict-F1 0.9961, but that split shares generators with training. evaluation/probes.json is the honest number: 8/10 on ten hand-written out-of-frame sentences.