eurovoc-eubert

EuroVoc Domain Classifier (EUBERT)

A fine-tuned EUBERT model that classifies EU legislation into the 21 top-level EuroVoc thematic domains.

EUBERT is the European Parliament's own BERT model, pre-trained from scratch on EU legislative texts rather than Wikipedia. This classifier adds a classification head fine-tuned on 64,000 labelled regulations.

Why this exists

The European Parliament published an official EuroVoc classifier on HuggingFace. It ships with mismatched vocabulary and weight dimensions and does not produce usable output. A community report confirmed the issue in 2024. No fix was published. This model is a working replacement.

Performance

Evaluated on 890 held-out EU regulations published between September 2025 and March 2026. Ground truth labels were assigned by professional librarians at the EU Publications Office.

Context: Series Results

Full write-up: Pimp My LM: A Fine-Tuning Tale of Bling and Basic

The 21 Labels

AGRI-FOODSTUFFS, AGRICULTURE FORESTRY AND FISHERIES,
BUSINESS AND COMPETITION, ECONOMICS, EDUCATION AND COMMUNICATIONS,
EMPLOYMENT AND WORKING CONDITIONS, ENERGY, ENVIRONMENT,
EUROPEAN UNION, FINANCE, GEOGRAPHY, INDUSTRY,
INTERNATIONAL ORGANISATIONS, INTERNATIONAL RELATIONS, LAW,
POLITICS, PRODUCTION TECHNOLOGY AND RESEARCH, SCIENCE,
SOCIAL QUESTIONS, TRADE, TRANSPORT

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_name = "jngb-labs/eurovoc-eubert"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)
model.eval()

text = "Regulation establishing a carbon border adjustment mechanism..."
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)

with torch.no_grad():
    logits = model(**inputs).logits
    probs = torch.sigmoid(logits)

LABELS = [
    "AGRI-FOODSTUFFS", "AGRICULTURE, FORESTRY AND FISHERIES",
    "BUSINESS AND COMPETITION", "ECONOMICS", "EDUCATION AND COMMUNICATIONS",
    "EMPLOYMENT AND WORKING CONDITIONS", "ENERGY", "ENVIRONMENT",
    "EUROPEAN UNION", "FINANCE", "GEOGRAPHY", "INDUSTRY",
    "INTERNATIONAL ORGANISATIONS", "INTERNATIONAL RELATIONS", "LAW",
    "POLITICS", "PRODUCTION, TECHNOLOGY AND RESEARCH", "SCIENCE",
    "SOCIAL QUESTIONS", "TRADE", "TRANSPORT"
]

threshold = 0.35
predictions = [LABELS[i] for i, p in enumerate(probs[0]) if p > threshold]
print(predictions)

Training Details

• Base model: EuropeanParliament/EUBERT (110M parameters, pre-trained on EU legislative texts)
• Training data: 63,918 EU regulations (preambles), sourced from EUR-Lex via CELLAR API
• Test data: 890 held-out regulations, labels assigned by EU Publications Office librarians
• Architecture: BERT pooler → Dropout(0.1) → Linear(768, 21) → Sigmoid
• Loss: BCEWithLogitsLoss
• Epochs: 3
• Hardware: Nvidia T4 (Google Colab)
• Training time: 159 minutes
• Cost: Free (Google Colab free tier)

Limitations

• Trained on English-language preambles only (EU legislation is published in 24 languages)
• Multi-label threshold (0.35) was optimised on the test set; may need adjustment for other corpora
• Classification granularity is limited to 21 top-level domains; finer EuroVoc concepts are not predicted

License

Apache 2.0

Author

Jakob Neugebauer