SciLaD-M-roberta

SciLaD-M (RoBERTa tokenizer)

SciLaD-M-roberta is a RoBERTa-base encoder pre-trained from scratch on the English split of the SciLaD corpus of scientific publications, using the original RoBERTa tokenizer (FacebookAI/roberta-base). This variant isolates the effect of the training corpus from the effect of the vocabulary, and makes the model drop-in compatible with pipelines that already tokenize with RoBERTa.

A sibling model, scilons/SciLaD-M-custom, uses a tokenizer trained from scratch on SciLaD and has a slightly higher average benchmark score; see Evaluation to decide which variant fits your use case.

Intended use

• Primary: fine-tuning on scientific-text tasks — NER (biomedical, CS), PICO extraction, relation classification, citation-intent and field-of-research classification, dependency parsing.
• Also: masked-token fill-in, contextual embeddings for retrieval or clustering of scientific documents, and drop-in replacement in existing RoBERTa-tokenized pipelines.
• Out of scope: text generation (this is an encoder, not a generative LM), instruction following, non-English text, high-stakes clinical/diagnostic use without further validation.

How to use

from transformers import AutoTokenizer, AutoModelForMaskedLM, pipeline

tok = AutoTokenizer.from_pretrained("scilons/SciLaD-M-roberta")
model = AutoModelForMaskedLM.from_pretrained("scilons/SciLaD-M-roberta")

fill = pipeline("fill-mask", model=model, tokenizer=tok)
fill(f"Transformer models achieve state-of-the-art {tok.mask_token} on many NLP benchmarks.")

For fine-tuning on downstream tasks, load the appropriate task head, e.g. AutoModelForTokenClassification for NER or AutoModelForSequenceClassification for classification:

from transformers import AutoModelForTokenClassification

model = AutoModelForTokenClassification.from_pretrained(
    "scilons/SciLaD-M-roberta", num_labels=NUM_LABELS
)

Training data

Pre-trained on the English clean plain-text split of SciLaD, published on the Hugging Face Hub as scilons/SciLaD-all-text-v1: 10.99M scientific publications (68.7B tokens, avg. 6k tokens/doc) harvested from open-access sources via an entirely open-source pipeline:

• Unpaywall snapshot (14 June 2023) → PDFs → TEI XML with Grobid
• arXiv → LaTeX → TEI XML with LaTeXML
• PMC / PLOS → JATS XML → TEI XML with Pub2TEI
• Language filtering (weighted fastText ≥ 0.8 English), Gopher quality heuristics, and MinHash near-duplicate removal (5-grams, 112 hash functions, 75% Jaccard threshold) via Datatrove.

See the paper for full dataset statistics and the open pipeline.

Training procedure

• Architecture: RoBERTa-base — 12 Transformer layers, ~110M parameters.
• Objective: Masked Language Modeling (MLM).
• Tokenizer: the original FacebookAI/roberta-base byte-pair tokenizer (vocabulary size 50,265), reused unchanged. Only the tokenizer is reused — the model weights are initialised randomly and trained entirely from scratch on SciLaD.
• Batching: batch size 256 × 2 gradient-accumulation steps → effective batch size 1024.
• Hardware: 2 × NVIDIA H200 80GB.
• Data split: 80 / 10 / 10 (train / validation / test) at document level.
• Hyperparameters: otherwise as in the original RoBERTa configuration.

Evaluation

Fine-tuned on 11 scientific-NLP benchmarks (5 task families); F1 unless noted, averaged over 3 seeds. A compact view of SciLaD-M-roberta vs. SciBERT and SciLaD-M-custom:

This variant is the highest-scoring of the compared models on NCBI-disease, EBM-NLP, and GENIA (UAS). Its overall average (73.55) is slightly below SciLaD-M-custom (74.91) but still ahead of SciBERT (71.19), RoBERTa (65.50), ModernBERT (65.51), and BERT (59.47). Full per-task numbers (including BERT, RoBERTa, and ModernBERT baselines with standard deviations) are in the paper.

When to prefer this variant

• You already have a tokenizer-dependent pipeline built around FacebookAI/roberta-base and want to swap in a scientific-domain model without retokenizing.
• You need PICO or NCBI-disease NER performance specifically (this variant is strongest there).

Otherwise, consider scilons/SciLaD-M-custom, which has the higher overall average.

Limitations and biases

• English only. The pre-training corpus was filtered to English; performance on other languages is not guaranteed.
• Domain skew. The open-access corpus is heavily biomedical / life-sciences, with smaller but meaningful coverage of physical sciences, CS, and social sciences. Expect stronger performance on biomedical text than on, e.g., pure-CS or humanities benchmarks (visible in the lower SciERC-NER score).
• Tokenizer mismatch. The tokenizer was built for general-domain RoBERTa, so scientific terms are often split into more sub-word pieces than necessary. This is the motivation for SciLaD-M-custom.
• No alignment. The model has not been instruction- or safety-tuned. It reflects the distribution of the scientific literature it was trained on, including any biases therein.
• Encoder only. Not suitable for open-ended generation.

License

Released under the Apache License 2.0.

Citation

If you use this model, please cite the accompanying paper:

@misc{foppiano2025scilad,
  title        = {SciLaD: A Large-Scale, Transparent, Reproducible Dataset for Natural Scientific Language Processing},
  author       = {Foppiano, Luca and Takeshita, Sotaro and Ortiz Suarez, Pedro and
                  Borisova, Ekaterina and Abu Ahmad, Raia and Ostendorff, Malte and
                  Barth, Fabio and Moreno-Schneider, Julian and Rehm, Georg},
  year         = {2025},
  eprint       = {2512.11192},
  archivePrefix= {arXiv},
  url          = {https://arxiv.org/abs/2512.11192}
}

Acknowledgements

Developed by researchers affiliated with DFKI, Humboldt-Universität zu Berlin, ScienciaLAB, Common Crawl Foundation, and the University of Mannheim. Supported by NFDI4DS (DFG project 460234259) and the EU Horizon Europe project HIVEMIND (No. 101189745). Compute provided by ACCESS / JetStream 2 (allocation CIS220172), TACC, and the University of Mannheim / DFKI.