InvDef-DeBERTa

InvDef-DeBERTa Model Card

The InvDef-DeBERTa is a transformer encoder model pretrained for the domain of invasion biology. In addition to MLM pretraining on scientific abstracts (ca. 35000) from the domain of invasion biology, we pretrain it as embedding model on concept definitions for domain-relevant concepts. This dataset of concepts with definitions was created using an LLM by first extracting concepts from the scientific abstracts and then generating definitions for them.

Model Details

Model Description

• Developed by: CLAUSE group at Bielefeld University
• Model type: DeBERTa-base
• Languages: Mostly english
• Finetuned from model: microsoft/deberta-base

Model Sources

• Repository: github.com/inas-argumentation/Ontology_Pretraining
• Paper: aclanthology.org/2025.findings-emnlp.1238/

How to Get Started with the Model

Minimal example on how to process texts with this model:

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("CLAUSE-Bielefeld/InvDef-DeBERTa")
model = AutoModel.from_pretrained("CLAUSE-Bielefeld/InvDef-DeBERTa")

text = "Your text to be embedded."
batch = tokenizer([text], return_tensors="pt")
model_output = model(**batch)

Training Details

This model was trained on a dataset of about 35000 scientific abstracts from the domain of invasion biology. Additionally, we used a dataset of 23,597 unique concepts extracted from the abstracts by an LLM, each accompanied by at least four LLM-generated concept definitions. We used a triplet loss to encourage definitions of the same concept to be placed nearby in the embedding space, and to also place related concepts (that co-occur frequently) in proximity. The dataset and exact training procedure can be found in our GitHub repo,

Evaluation

The better-performing InvOntDef-DeBERTa was also trained by us, using ontology-derived data instead of purely LLM-generated data.

Citation

BibTeX:

@inproceedings{brinner-etal-2025-enhancing,
    title = "Enhancing Domain-Specific Encoder Models with {LLM}-Generated Data: How to Leverage Ontologies, and How to Do Without Them",
    author = "Brinner, Marc Felix  and
      Al Mustafa, Tarek  and
      Zarrie{\ss}, Sina",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.findings-emnlp.1238/",
    doi = "10.18653/v1/2025.findings-emnlp.1238",
    pages = "22740--22754",
    ISBN = "979-8-89176-335-7"
}