XLM-RoBERTa-CRF-VotIE

XLM-RoBERTa-CRF-VotIE: Portuguese Voting Information Extraction

This model is a fine-tuned XLM-RoBERTa Base with a Conditional Random Fields (CRF) layer for extracting structured voting information from Portuguese municipal meeting minutes. It achieves strong performance on the Citilink dataset.

Model Description

XLM-RoBERTa-CRF-VotIE combines Facebook AI's multilingual XLM-RoBERTa encoder with a CRF layer for structured sequence prediction. The model performs token-level classification to identify and extract voting-related entities from Portuguese administrative text.

Key Features

• Architecture: XLM-RoBERTa Base (768-dim, 12 layers, 12 heads, absolute positional attention) + Linear + CRF
• Task: Sequence Labeling with BIO tagging
• Language: Portuguese (Portugal)
• Domain: Municipal meeting minutes and voting records
• Entity Types: 12 types (25 labels with BIO encoding, full B/I pairs for all types)
• Performance: 95.28% entity-level F1 score

Intended Uses

This model is designed for:

• Extracting voting information from Portuguese municipal documents
• Identifying participants and their voting positions (favor, against, abstention, absent)
• Recognizing voting subjects and counting methods
• Structuring unstructured administrative text
• Research in information extraction from Portuguese administrative documents

Entity Types

The model recognizes 12 entity types in BIO format (25 labels total, full B/I pairs for every type):

Training Details

Training Data

The model was trained on the Citilink dataset (https://rdm.inesctec.pt/dataset/cs-2025-007), which consists of Portuguese municipal meeting minutes annotated with voting information:

• Training set: 1,737 examples
• Validation set: 433 examples
• Test set: 529 examples
• Total tokens: ~300K tokens
• Total entities: ~5K entities

Training Procedure

Hyperparameters:

• Base model: FacebookAI/xlm-roberta-base
• Batch size: 16
• Learning rate: 5e-5 (linear decay)
• Warmup proportion: 0.1
• Max grad norm: 1.0
• Dropout: 0.1
• Max sequence length: 512 tokens
• Epochs: 10 (best checkpoint at epoch 7)
• Optimizer: AdamW
• Early stopping patience: 3 epochs
• Loss: CRF negative log-likelihood

Results

Entity-Level Performance (Test Set, 529 examples)

Per-Entity Performance (Strict)

Usage

Quick Start

The simplest way to use the model:

from transformers import AutoTokenizer, AutoModel

# Load model
model_name = "Anonymous3445/XLM-RoBERTa-CRF-VotIE"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True)

# Analyze text
text = "O Executivo deliberou aprovar o projeto por unanimidade."
inputs = tokenizer(text, return_tensors="pt")
predictions = model.decode(**inputs, tokenizer=tokenizer, text=text)

# Print results
for pred in predictions:
    print(f"{pred['word']:20} {pred['label']}")

Output:

O                    B-VOTER-FAVOR
Executivo            I-VOTER-FAVOR
deliberou            B-VOTING
aprovar              O
o                    O
projeto              O
por                  B-COUNTING-UNANIMITY
unanimidade.         I-COUNTING-UNANIMITY

Extract Structured Entities

The model includes a convenient extract_entities method:

from transformers import AutoTokenizer, AutoModel

model_name = "Anonymous3445/XLM-RoBERTa-CRF-VotIE"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True)

text = """A Câmara Municipal deliberou aprovar a proposta apresentada pelo
Senhor Presidente. Votaram a favor os Senhores Vereadores João Silva e
Maria Costa. Votou contra o Senhor Vereador Pedro Santos."""

# Get structured entities with character offsets
entities = model.extract_entities(text, tokenizer)

# Print entities by type
for entity_type, mentions in entities.items():
    print(f"\n{entity_type}:")
    for mention in mentions:
        print(f"  - {mention['text']} [{mention['start']}:{mention['end']}]")

Output:

VOTER-FAVOR:
  - A Câmara Municipal [0:19]
  - João Silva [95:105]
  - Maria Costa [108:119]

VOTING:
  - deliberou [20:29]

VOTER-AGAINST:
  - Pedro Santos [152:164]

Token-Level Predictions

For low-level token-based predictions:

from transformers import AutoTokenizer, AutoModel

model_name = "Anonymous3445/XLM-RoBERTa-CRF-VotIE"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True)

text = "O Executivo deliberou aprovar por unanimidade."
inputs = tokenizer(text, return_tensors="pt")

# Get raw token-level predictions (list of label IDs)
predictions = model.decode(inputs["input_ids"], inputs["attention_mask"])
# Returns: [[0, 7, 15, 8, 0, 0, 2, 10, 0]]

# Convert to labels
id2label = model.config.id2label
tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])

for token, pred_id in zip(tokens, predictions[0]):
    if token not in ['<s>', '</s>', '<pad>']:
        print(f"{token:20} {id2label[pred_id]}")

With Character Offsets

Useful for highlighting entities in your UI:

from transformers import AutoTokenizer, AutoModel

model_name = "Anonymous3445/XLM-RoBERTa-CRF-VotIE"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True)

text = "O Executivo deliberou aprovar o projeto por unanimidade."
inputs = tokenizer(text, return_tensors="pt")

# Get predictions with character positions
predictions = model.decode(**inputs, tokenizer=tokenizer, text=text, return_offsets=True)

# Show only entities (non-O tags)
for pred in predictions:
    if pred['label'] != 'O':
        print(f"{pred['word']:20} {pred['label']:25} [{pred['start']}:{pred['end']}]")

Output:

O                    B-VOTER-FAVOR             [0:1]
Executivo            I-VOTER-FAVOR             [2:11]
deliberou            B-VOTING                  [12:21]
por                  B-COUNTING-UNANIMITY      [39:42]
unanimidade.         I-COUNTING-UNANIMITY      [43:55]

Batch Processing

For processing multiple documents:

from transformers import AutoTokenizer, AutoModel

model_name = "Anonymous3445/XLM-RoBERTa-CRF-VotIE"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True)

texts = [
    "A proposta foi aprovada por unanimidade.",
    "Votou contra o Vereador João Silva.",
    "O Presidente estava ausente na votação."
]

for text in texts:
    entities = model.extract_entities(text, tokenizer, return_offsets=False)
    print(f"\nText: {text}")
    for entity_type, mentions in entities.items():
        print(f"  {entity_type}: {[m['text'] for m in mentions]}")

Limitations and Bias

Limitations

• Domain-specific: Trained specifically on Portuguese municipal meeting minutes; may not generalize well to other document types
• Portuguese only: Optimized for European Portuguese
• Sequence length: Limited to 512 tokens per window (handles longer documents via windowing)
• Entity types: Limited to 12 predefined voting-related entity types
• Complex sentences: May struggle with highly complex or nested voting descriptions

Requirements

Install the required dependencies:

pip install torch>=2.0.0 transformers>=4.30.0 pytorch-crf>=0.7.2

Model Card Authors

• Anonymous Authors (for blind review)

Model Card Contact

For questions or issues, please open an issue in the GitHub repository.

Additional Resources

• GitHub Repository: https://github.com/Anonymous3445/VotIE
• Dataset: Citilink Dataset
• Paper: [Coming soon]
• Demo: VotIE Demo

License

This model is released under the Creative Commons Attribution-NoDerivatives 4.0 International (CC BY-ND 4.0) license.

• ✅ You can: Use the model for research and commercial purposes with attribution
• ❌ You cannot: Create derivative works or modified versions
• 📝 You must: Provide attribution to the original authors

See LICENSE for full details.

Acknowledgments

This work builds upon:

• XLM-RoBERTa: Facebook AI's multilingual XLM-RoBERTa base model
• pytorch-crf: CRF implementation
• Transformers: Hugging Face Transformers library

---

Version: 4.0
Last Updated: 2026-06-02