xlm-roberta-NER-syntheticGov

Estonian NER Model - Fine-tuned on Chat-Style Data

This model is a domain-adapted version of FacebookAI/xlm-roberta-large-finetuned-conll03-english, further fine-tuned on Estonian chat-style text with synthetic data augmentation.

Model Description

• Base Model: FacebookAI/xlm-roberta-large-finetuned-conll03-english
• Language: Estonian (et) — multilingual encoder pretrained on 100 languages
• Task: Token Classification (Named Entity Recognition)
• Training Data: Hand-annotated Estonian chat conversations augmented with synthetic data

This model specializes in extracting named entities from Estonian chat-style text — short, informal communications such as citizen-to-bot conversations, customer service chats, and similar interactive dialogue. The base encoder (XLM-RoBERTa-large) was already fine-tuned on CoNLL-2003 English NER by Facebook, giving it a strong NER inductive prior before our Estonian fine-tune.

Supported Entity Types

The model recognizes 4 entity types:

• PER: Person names
• ORG: Organizations, companies, government agencies
• LOC: Locations, addresses, streets, buildings
• GPE: Geopolitical entities (cities, counties, countries)

Each entity uses BIO tagging (B- for beginning, I- for inside), giving 9 total label classes (4 × 2 + O).

Training Data

The model was fine-tuned on a combination of hand-annotated Estonian chat data and synthetically generated examples designed to improve coverage of person and organization entities.

• Generation Method: Synthetic examples generated using Google Gemini-3-pro API with structured prompts
• Domain Coverage: Estonian chat / citizen-service communications, with deliberate diversity in writing style (formal, casual, shorthand, mixed)
• Topics: Conversations about Estonian government services, addresses, persons, organizations, and locations in Estonia and neighbouring regions

Why Synthetic Data?

Synthetic data generation allowed us to:

• Address class imbalance in the real chat data, where PER and ORG entities were under-represented
• Generate additional diverse name and organization examples without requiring more human annotation
• Include linguistic variations (declensions of person and organization names in Estonian) that the small real corpus did not fully cover
• Control entity distribution

Training Details

• Base Model: FacebookAI/xlm-roberta-large-finetuned-conll03-english (~559 M params, XLM-RoBERTa-large architecture)
• Training Epochs: up to 30, with early stopping (patience 5 on dev F1)
• Best Epoch: 4 (dev F1 = 0.995); total training time ≈ 649 s on a single 11 GB GPU
• Batch Size: 2 (effective 16 with gradient accumulation = 8)
• Learning Rate: 3e-5
• Max Sequence Length: 128 tokens (memory-constrained; the model itself supports 512)
• Optimizer: AdamW (weight decay: 0.01)
• Warmup Ratio: 0.1
• Precision: fp16 mixed precision
• Memory Optimization: gradient checkpointing enabled to fit the 559 M-param model on 11 GB
• Head Initialization: original 8-label CoNLL-2003 (IOB1) head was discarded with ignore_mismatched_sizes=True; a fresh 9-label IOB2 head was trained from scratch
• Training Framework: Hugging Face Transformers + PyTorch

Usage

from transformers import AutoTokenizer, AutoModelForTokenClassification
from transformers import pipeline

# Load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained('buerokratt/{model_name}')
model = AutoModelForTokenClassification.from_pretrained('buerokratt/{model_name}')

# Create NER pipeline
nlp = pipeline("ner", model=model, tokenizer=tokenizer, aggregation_strategy="simple")

# Example text
text = "Tere, mind kutsutakse Mati Tamm ja ma elan Tallinnas Pirita teel."

# Get predictions
ner_results = nlp(text)
for entity in ner_results:
    print(f"{entity['word']}: {entity['entity_group']} ({entity['score']:.3f})")

Overall Metrics

Evaluated on data/test_fresh.conll — 70 hand-checked chat sentences with no entity-name overlap with the training set (so the numbers reflect true generalization, not vocabulary memorization).

Per-Entity Performance

Intended Use

This model is optimized for:

• Processing Estonian chat-style citizen-service conversations
• Extracting entities from short, informal Estonian text
• Information extraction from interactive dialogue (bot conversations, support chats)
• Estonian named-entity tagging in mixed-style text (formal + casual + shorthand)

Limitations

• Domain Specificity: Optimized for chat-style text; performs less well on long-form news (≈ 0.74 F1 on EstNER v1 vs 0.87 on chat-fresh). Use a different model or a mixed fine-tune for news.
• GPE class is the weakest (F1 0.75 on fresh test) — many smaller Estonian place names are still confused with LOC/ORG when the surrounding context is ambiguous.
• Synthetic Training Data: While diverse, synthetic data may not capture every real-world linguistic variation. The synthetic portion was specifically designed to balance PER/ORG coverage and may bias the model toward common name patterns.
• IOB1 Origin: The base model was originally trained on CoNLL-2003 in the IOB1 scheme. The fine-tuned head used here is IOB2 (every entity starts with B-), so this is not an issue at inference, but be aware when comparing label distributions to the base model's outputs.

Citation

If you use this model, please cite the base XLM-RoBERTa paper:

@article{conneau2019unsupervised,
  title={Unsupervised Cross-lingual Representation Learning at Scale},
  author={Conneau, Alexis and Khandelwal, Kartikay and Goyal, Naman and Chaudhary, Vishrav and Wenzek, Guillaume and Guzm{\'a}n, Francisco and Grave, Edouard and Ott, Myle and Zettlemoyer, Luke and Stoyanov, Veselin},
  journal={arXiv preprint arXiv:1911.02116},
  year={2019}
}

Acknowledgments

• Base Model: FacebookAI/xlm-roberta-large-finetuned-conll03-english by Meta AI Research (Facebook AI), fine-tuned on CoNLL-2003
• Synthetic Data Generation: Google Gemini-3-pro API
• Training Framework: Hugging Face Transformers