19c_roberta_v5_newspaper

19c RoBERTa v5 Newspaper Language Model (In Development)

This repository is set up for the upcoming RoBERTa v5 language model, which will be pretrained on a newly reconstructed, high-resolution 19th-century American newspaper corpus.

The Core Strategy: Replacing Noisy OCR

The existing Tesseract OCR from Chronicling America (loc.gov) suffers from error rates of 12–40% due to degraded microfilms, mixed case, long-s (ſ) ligatures, and poor line segmentation.

To build a clean corpus for RoBERTa v5, we are bypassing the bad original OCR and performing OCR reconstruction from the original page images using a custom pipeline:

1. High-Resolution Harvest: Digitized pages downloaded at 50% resolution (doubling linear pixel density for small body text).
2. YOLO Column Segmentation: A fine-tuned YOLO detector segments the newspaper pages into clean column crops.
3. Strip Tiling: Slices columns into overlapping horizontal strips to keep text density within VLM visual attention boundaries.
4. Local VLM Transcription: Transcribes strips using Gemma MoE via LM Studio APIs (achieving 90–95% transcription accuracy).
5. Deduplication & Merge: Reassembles transcripts using sequence matching and de-hyphenation.
6. Quality Gating via RoBERTa v4:
   • Perplexity (PPL) Filtering: Assembled column texts are evaluated by the RoBERTa v4 language model. High-perplexity columns (indicating gibberish, heavy distortion, or transcription failure) are automatically discarded.
   • Looping & Hallucination Mitigation: Microfilm defects or visual artifacts can cause the VLM to repeat paragraphs (looping), output text in blank areas, or generate plausible but hallucinated paragraphs. RoBERTa v4's perplexity scores easily flag and filter out these repeated blocks, empty segments, and anomalies.

---

Pretraining Corpus: The 19c Newspaper Harvest

The source corpus has been harvested and banked in ambrosfitz/19c_newspapers_images_alto containing 1,361 Parquet shards (~68,050 images).

Decade-by-Decade Banked Image Counters

---

How to Load (Once Released)

Once training is complete, the model will be loadable using the Hugging Face transformers library:

from transformers import AutoModelForMaskedLM, AutoTokenizer

repo_id = "ambrosfitz/19c_roberta_v5_newspaper"

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForMaskedLM.from_pretrained(repo_id)