pii-modernbert-110m-mlm-3b

PII ModernBERT 110M MLM 3B Warmup

This is the encoder warmup checkpoint for the open-source PII_MODEL browser-local redaction project. It is a custom PyTorch ModernBERT-style masked-language-model checkpoint intended as the base for the final span-level PII token classifier.

Important Status

• This is not yet the final PII redaction classifier.
• It is a masked-language-model warmup checkpoint trained before supervised PII span fine-tuning.
• The architecture is custom and should be loaded with the pii_model.modeling code from the project repository.

Training Summary

• Hub repo: soumikmahato/pii-modernbert-110m-mlm-3b
• Tokenizer: soumikmahato/pii-redactor-tokenizer-v1
• Tokens seen: 3000008704
• Parameters: 109408512
• Final logged loss: 3.3695396184921265
• Final logged EMA loss: 2.989036085762805
• Context length: 8192
• Hidden size/layers/heads: 768 / 12 / 12
• Local attention window: 512
• Global attention every N layers: 3

Files

• model.safetensors: model-only state dict from the final MLM checkpoint.
• config.json: custom PiiEncoderConfig.
• training_state.json: final trainer state and corpus summary.
• summary.json: training summary emitted by the trainer.
• metrics.jsonl: logged training metrics.
• checkpoint.pt: optional raw trainer checkpoint if included in this export.

Intended Use

Use this checkpoint to initialize supervised PII span classification training. Do not ship this warmup model alone as a redactor.

Privacy Boundary

The target product is a local Chrome extension that redacts sensitive spans before prompts are sent to AI chatbots. Raw user text and redaction maps should remain on-device.