xlm-roberta-base-ess-classification

XLM-RoBERTa-Base for ESS Variable Classification

Fine-tuned XLM-RoBERTa-Base model for classifying European Social Survey variables into 19 subject categories.

Model Description

This model is a fine-tuned version of FacebookAI/xlm-roberta-base on the ESS variable classification dataset. It classifies survey questions and variables into predefined subject categories.

• Base Model: XLM-RoBERTa-Base (125M parameters)
• Task: Multi-class text classification (19 categories)
• Language: English
• Dataset: European Social Survey variables

Performance

Evaluated on test set:

• Accuracy: 0.8381
• Precision (weighted): 0.7858
• Recall (weighted): 0.8381
• F1-Score (weighted): 0.7959
• Test samples: 105

Intended Use

This model is designed to automatically classify survey variables and questions from social science research into subject categories. It can be used for:

• Organizing large survey datasets
• Automating metadata generation
• Subject classification of research questions
• Data cataloging and discovery

Training Data

The model was trained on the benjaminBeuster/ess_classification dataset, which contains survey variables extracted from European Social Survey DDI XML files.

Label Mapping

The model predicts one of 19 subject categories:

Usage

Basic Classification

from transformers import AutoModelForSequenceClassification, AutoTokenizer, pipeline

# Load model and tokenizer
model_name = "benjaminBeuster/xlm-roberta-base-ess-classification"
model = AutoModelForSequenceClassification.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Create classification pipeline
classifier = pipeline("text-classification", model=model, tokenizer=tokenizer)

# Classify a survey question
text = "Trust in country's parliament. Using this card, please tell me on a score of 0-10 how much you personally trust each of the institutions I read out."
result = classifier(text)

print(f"Category: {result[0]['label']}")
print(f"Confidence: {result[0]['score']:.4f}")

Batch Classification

# Classify multiple questions
questions = [
    "How often pray apart from at religious services",
    "Highest level of education completed",
    "Trust in politicians"
]

results = classifier(questions)
for question, result in zip(questions, results):
    print(f"{question[:50]}: {result['label']} ({result['score']:.2f})")

Manual Prediction

import torch

# Tokenize input
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=256)

# Get predictions
with torch.no_grad():
    outputs = model(**inputs)
    predictions = torch.nn.functional.softmax(outputs.logits, dim=-1)
    predicted_class = torch.argmax(predictions, dim=-1).item()

# Get label name
label = model.config.id2label[predicted_class]
confidence = predictions[0][predicted_class].item()

print(f"Predicted: {label} (confidence: {confidence:.4f})")

Training Procedure

Training Hyperparameters

• Learning rate: 2e-05
• Batch size: 8
• Epochs: 5
• Weight decay: 0.01
• Warmup ratio: 0.1
• Max sequence length: 256
• Optimizer: AdamW
• LR scheduler: Linear with warmup

Training Details

The model was fine-tuned using the Hugging Face Transformers library with the following setup:

• Early stopping with patience of 2 epochs
• Evaluation on validation set after each epoch
• Best model selection based on validation loss
• Mixed precision training (fp16/bf16 where supported)

Limitations and Bias

• The model is trained on a relatively small dataset (50 samples), which may limit generalization
• Performance may vary on survey questions outside the European Social Survey domain
• The model may inherit biases present in the training data
• English-language surveys are the primary focus, though the base model supports 100 languages

Citation

If you use this model, please cite:

@misc{xlm-roberta-ess-classifier,
  author = {Benjamin Beuster},
  title = {XLM-RoBERTa-Large for ESS Variable Classification},
  year = {2025},
  publisher = {Hugging Face},
  url = {https://huggingface.co/benjaminBeuster/xlm-roberta-base-ess-classification}
}

Model Card Authors

Benjamin Beuster

Model Card Contact

For questions or feedback, please open an issue on the model repository.