wikiann-multilingual-ner-mdeberta

WikiANN Multilingual NER (mDeBERTa-v3-base)

This model is a robust Multilingual Named Entity Recognition (NER) model fine-tuned on the WikiANN (PAN-X) benchmark across 7 major languages.

Built on top of microsoft/mdeberta-v3-base, it provides highly accurate extraction for the core classic NER entities: Person (PER), Organization (ORG), and Location (LOC) (7 BIO output logits).

Developed by Rishabh Kumar.

---

1. Model Description & Architecture

• Base Model: microsoft/mdeberta-v3-base (278M parameters, 12 layers, 768 hidden size, 250k multilingual vocabulary)
• Fine-Tuning Method: LoRA (r=32, alpha=32, target modules: query_proj, key_proj, value_proj, intermediate.dense, output.dense) merged into standalone base weights.
• Precision: FP16 (torch.float16, ~530 MB footprint)
• Task: Multilingual Classic Token Classification / NER
• Languages Supported (7): English (en), German (de), Spanish (es), French (fr), Italian (it), Portuguese (pt), Swedish (sv).

---

2. Dataset & Preprocessing Pipeline

A. Source Data & Splits

• Source: WikiANN (PAN-X) benchmark (unimelb-nlp/wikiann).
• Train Set: 140,000 sentences (20k per language across 7 languages).
• Validation Set: 70,000 sentences (10k per language).
• Test Set: 70,000 sentences (10k per language).

B. Preprocessing & Formatting

1. CoNLL Standardization: Raw PAN-X splits are formatted into clean 2-column CoNLL format (token tag) with sentence boundary separation.
2. Subword Token Alignment: Aligned with mDeBERTa-v3-base SentencePiece subword tokenizer with add_prefix_space=True. Leading subwords receive the ground-truth BIO tag (B-PER, B-LOC, B-ORG), and following subwords are masked with -100.
3. Tag Space: Clean 7-tag output vocabulary (O, B-PER, I-PER, B-LOC, I-LOC, B-ORG, I-ORG).

---

3. Entity Taxonomy (3 Classes / 7 BIO Tags)

---

4. Training Results & Metrics

The model was fine-tuned for 5 epochs (6,250 steps) using AdamW (lr=2e-5, linear scheduler, effective batch size 16).

Validation Performance (Best Checkpoint - Epoch 5)

Per-Epoch Progression

---

5. How to Use

Quick Inference via Hugging Face Pipeline

from transformers import pipeline

ner = pipeline(
    "token-classification",
    model="Rishabh157/wikiann-multilingual-ner-mdeberta",
    aggregation_strategy="simple"
)

text = "Barack Obama visited the United Nations headquarters in New York City."
entities = ner(text)

for entity in entities:
    print(f"{entity['word']:<25} | {entity['entity_group']:<15} | Score: {entity['score']:.4f}")

PyTorch Direct Usage

import torch
from transformers import AutoTokenizer, AutoModelForTokenClassification

model_name = "Rishabh157/wikiann-multilingual-ner-mdeberta"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForTokenClassification.from_pretrained(model_name)

text = "Albert Einstein wurde in Ulm geboren und lebte in Bern."
inputs = tokenizer(text, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)
    predictions = torch.argmax(outputs.logits, dim=2)

tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])
labels = [model.config.id2label[p.item()] for p in predictions[0]]

for token, label in zip(tokens, labels):
    if label != "O":
        print(f"{token:<20} -> {label}")

---

6. Citation

@misc{kumar2026wikiann,
  author = {Rishabh Kumar},
  title = {WikiANN Multilingual NER with mDeBERTa-v3},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Rishabh157/wikiann-multilingual-ner-mdeberta}}
}