mmbert-multilingual-impresso-continued-mlm

multilingualmodernimpressoBERT

Source card for the future Hugging Face model repository:

impresso-project/mmbert-multilingual-impresso-continued-mlm

This model is a continued-MLM adaptation of jhu-clsp/mmBERT-base on multilingual Impresso newspaper text.

The checkpoint is intended as a domain-adapted base model for downstream media-agency token classification.

License

This continued-pretraining checkpoint is released under the MIT license, matching the license of the base model jhu-clsp/mmBERT-base.

The model is derived from jhu-clsp/mmBERT-base; users should cite and comply with the base model terms. The continued-MLM training corpus consists of Impresso newspaper text samples used for domain adaptation. The model weights are published separately from the source text; users remain responsible for checking whether their downstream use of Impresso-derived models and outputs is compatible with their institutional, corpus, and application-specific requirements.

Continued MLM Run

The first completed workbench run used:

• source compiled Impresso files: fr, de, en, lb
• source filtering: OCR quality at least 0.90, minimum text length 200 characters
• source sampling policy: up to 300,000 texts per language, with smaller languages exhausted
• sampled corpus: 872,889 train rows and 8,818 validation rows after split
• training subset: 100,000 train rows
• validation subset: 2,000 validation rows
• base model: jhu-clsp/mmBERT-base
• objective: masked language modeling
• MLM probability: 0.15
• max sequence length: 256
• padding: fixed max-length padding
• epochs: 1
• per-device train batch size: 1
• gradient accumulation steps: 8
• effective train batch size per device: 8
• gradient checkpointing: enabled
• learning rate: 2e-5
• weight decay: 0.01
• warmup: 750 steps, computed as 6 percent of the capped optimizer steps
• intermediate checkpoint saving: disabled; final model only
• random seed: 42

Training was run locally on Apple Silicon using the PyTorch MPS backend. The completed run took about 8 hours and 6 minutes. Observed memory use stayed below roughly 30 GB.

Metrics

Final run metrics:

The validation loss stayed close to the train loss and the run completed without divergence, so this checkpoint is suitable for downstream comparison against the original jhu-clsp/mmBERT-base.