EuroBERT-610m-term-sense

T-EBERT: German IR Term Sense Disambiguation

Fine-tuned EuroBERT-610M for disambiguating International Relations term senses in German.

Model Description

This model distinguishes between specialized (International Relations) and colloquial usage of German political science terms:

Input: "Die internationale Norm verbietet den Einsatz von Gewalt."
Output: IR sense (specialized usage)

Input: "Das entspricht nicht den technischen Normen in Deutschland."
Output: Colloquial sense (general usage)

Performance

• F1 Score: 0.922 on held-out test set
• Training / Test distribution: 70% IR sense, 30% colloquial (natural corpus distribution) | Term | F1 Score | |------|----------| | Entspannung | 0.967 | | Intervention | 0.949 | | Integration | 0.933 | | Kooperation | 0.926 | | Norm | 0.831 | | Regime | 0.828 |

Intended Use

• German IR corpus analysis and text mining
• Political science document classification
• Automated terminology extraction in specialized texts
• Production deployment on naturally distributed data

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

# Load model and tokenizer
model_name = "pdjohn/EuroBERT-610m-term-sense"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

# Classify a sentence
sentence = "Die internationale Norm verbietet den Einsatz von Gewalt."
inputs = tokenizer(sentence, return_tensors="pt", truncation=True, padding=True)

with torch.no_grad():
    outputs = model(**inputs)
    prediction = torch.argmax(outputs.logits, dim=1).item()

# Interpretation
labels = {0: "Colloquial", 1: "IR Sense"}
print(f"Prediction: {labels[prediction]}")

Training Data

• Corpus: German political science texts from IR literature
• Size: 934 sentences (training) + 234 sentences (test)
• Distribution: 70% IR sense, 30% colloquial (natural)
• Terms: Norm, Kooperation, Regime, Integration, Intervention, Entspannung
• Annotation: Manual expert annotation

Training Procedure

• Base model: EuroBERT-610M
• Fine-tuning method: LoRA (Low-Rank Adaptation)
  • Rank (r): 8
  • Alpha: 16
  • Target modules: q_proj, k_proj, v_proj
  • Dropout: 0.05
• Epochs: 5
• Batch size: 8
• Learning rate: 1e-4
• Optimizer: AdamW with cosine learning rate schedule

Limitations

• Trained on imbalanced data (70/30 split)
• Performance may drop on datasets with different distributions
• see T-EBERT-Balanced for a variant with distribution robustness

Acknowledgements

This work was conducted as part of the Tracing International Institutions and Behavior (TIIB) project at TU Darmstadt.

Affiliation: Chair of Transnational Governance, Department of Political Science, Technische Universität Darmstadt

We thank the TIIB project team for their support and the manual annotation of the training corpus.