Downloads · 30 days
8
13% of all-time downloads
19c-modernbert-mini-b-news-gated-v2
ambrosfitz/19c-modernbert-mini-b-news-gated-v2
19c-modernbert-mini-b-news-gated-v2 is a fill-mask model from ambrosfitz. Use it when you need the model to fill a missing word. It is set up for transformers. The card lists the license as mit.
Continued MLM pretraining of 19c-modernbert-mini-b on the quality-filtered newspaper corpus, v2: full-corpus chunked packing.
All-time downloads
64
Public
Parameters
58.7M
117 MB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors117 MB · 98%
At a glance
- Task
- Fill-Mask
- Library
- transformers
- License
- mit
- Model type
- modernbert
- Access
- Public
- Created
- Jul 1, 2026
- Updated
- Jul 1, 2026
- SHA
- fc84022d
Try a prompt
Base models
Model card
From the Hugging Face model README
19c-ModernBERT-mini-B — Newspaper Gated v2 (full-corpus chunked)
Continued MLM pretraining of 19c-modernbert-mini-b on the quality-filtered newspaper corpus, v2: full-corpus chunked packing.
This supersedes
19c-modernbert-mini-b-news-gated.
The v1 run silently trained on ~16% of the corpus — truncation=True, max_length=512
turned each article into at most one 512-token sequence, discarding everything past
token 512. With articles averaging ~6,334 tokens, only the first ~8% of each article
survived, exposing the model to ~42M of 264M available tokens. This run fixes that
with standard group_texts packing: concatenate the full token stream and split into
contiguous 512-token blocks, wasting almost nothing.
What changed from v1
| v1 (gated) | v2 (this model) | |
|---|---|---|
| Tokenization | truncation=True, max_length=512 | No truncation |
| Packing | 1 sequence per document (truncated) | group_texts — contiguous 512-token blocks |
| Training sequences | ~41k | ~510k |
| Tokens seen | ~42M (~16% of corpus) | ~261M (~99% of corpus) |
| Steps (2 epochs) | 646 | 7,968 |
| Runtime | ~8 min | ~2 hr 1 min |
| Final PPL | 6.00 | 5.44 (best ckpt: 5.36) |
Curriculum position
19c-modernbert-mini-b <- from scratch on Gutenberg prose (PPL 5.05)
|
+-> 19c-modernbert-mini-b-news <- full OCR corpus, truncated
+-> 19c-modernbert-mini-b-news-gated <- gated OCR corpus, truncated (PPL 6.00)
+-> 19c-modernbert-mini-b-news-gated-v2 <- THIS MODEL — gated, full-corpus (PPL 5.44)
Training
| Base checkpoint | ambrosfitz/19c-modernbert-mini-b (58.7M params) |
| Corpus | ambrosfitz/ca_19c_ocr_combined_gated — 41,669 docs, ~264M tokens |
| Objective | MLM, 15% masking |
| Packing | group_texts across batch (blocks may straddle article boundaries — standard for MLM pretraining) |
| Train blocks | 509,952 x 512 tokens |
| Epochs | 2 |
| Steps | 7,968 |
| Effective batch size | 128 (batch 64 x grad_accum 2) |
| Learning rate | 5e-5, linear decay, 6% warmup |
| Weight decay | 0.01 |
| Precision | bf16 |
| Attention | SDPA |
| Hardware | RunPod RTX A6000 (48GB) |
| Runtime | 2 hr 1 min |
Evaluation (1% held-out document split, chunked)
| Step | Epoch | Eval Loss | PPL |
|---|---|---|---|
| 0 (baseline) | — | 3.8796 | 48.40 |
| 500 | 0.13 | 1.825 | 6.21 |
| 1000 | 0.25 | 1.749 | 5.75 |
| 2000 | 0.50 | 1.701 | 5.48 |
| 4000 | 1.00 | 1.686 | 5.41 |
| 5500 | 1.38 | 1.690 | 5.42 |
| 7500 | 1.88 | 1.679 | 5.36 — best checkpoint |
| 7968 (final) | 2.00 | 1.692 | 5.44 |
Model saved from best checkpoint (step 7500, PPL 5.36). Loss plateaus around epoch 1.0 and remains stable through epoch 2, indicating the model has extracted most of the available signal at this LR and data scale.
Architecture
Inherits the base model architecture unchanged:
| Params | 58.7M |
| Hidden size | 576 |
| Layers | 12 |
| Attention | RoPE, alternating global / local (window 128) |
| Vocab | 32,101 (includes year tokens <1800>–<1900>) |
Loading
from transformers import AutoModelForMaskedLM, AutoTokenizer
model = AutoModelForMaskedLM.from_pretrained("ambrosfitz/19c-modernbert-mini-b-news-gated-v2")
tokenizer = AutoTokenizer.from_pretrained("ambrosfitz/19c-modernbert-mini-b-news-gated-v2")
Intended use and limitations
Encoder for 19th-century newspaper OCR text — NER, classification, embedding, span extraction. Not instruction-tuned; not for generation. Recommended checkpoint for downstream tasks on 19c newspaper text.
- Task
- Fill-Mask
- Library
- transformers
- Type
- modernbert
- License
- mit
- Languages
- en
- Created
- Jul 1, 2026
- Updated
- Jul 1, 2026