berturk-milliyetner-ner

BERTurk Fine-Tuned on MilliyetNER for Named Entity Recognition

This repository provides a BERTurk model fine-tuned on the MilliyetNER dataset for Turkish Named Entity Recognition (NER).

The model was fine-tuned only on contemporary Turkish data from MilliyetNER. It was not fine-tuned on HisTR or Ruznamçe. Historical Turkish datasets were used only for evaluation to measure zero-shot cross-domain transfer performance.

Model Details

Label Set

The model uses BIO-formatted entity labels.

Entity types:

• PER: Person
• LOC: Location
• ORG: Organization
• O: Outside a named entity

Training Strategy

The model was fine-tuned only on the MilliyetNER training set.

No Historical Turkish examples from HisTR or Ruznamçe were used during fine-tuning.

The evaluation design was:

1. Fine-tune BERTurk on the MilliyetNER training set.
2. Evaluate on the MilliyetNER development set.
3. Evaluate directly on the HisTR in-domain test set.
4. Evaluate directly on the Ruznamçe out-of-domain test set.

The HisTR and Ruznamçe results therefore represent zero-shot transfer from contemporary Turkish Named Entity Recognition to Historical Turkish Named Entity Recognition.

Training Configuration

Run Selection

Three runs were trained using seeds 42, 52, and 62.

For this Hugging Face release, the run with the highest strict entity-level overall F1 score on the HisTR in-domain test set was selected, following the release criterion specified for this study.

The released model is therefore the run trained with seed 62.

Evaluation

Evaluation was performed using strict entity-level matching. An entity was counted as correct only when both its span boundaries and entity type matched the gold annotation.

MilliyetNER Development Set

Overall Results

Entity-Level Results

HisTR In-Domain Test Set

Overall Results

Entity-Level Results

Ruznamçe Out-of-Domain Test Set

Overall Results

Entity-Level Results

Usage

Using the Transformers Pipeline

Usage

Using the Transformers Pipeline

from transformers import pipeline

ner = pipeline(
    task="token-classification",
    model="BUCOLIN/berturk-milliyetner-ner",
    aggregation_strategy="simple",
)

text = "Mustafa Kemal Ankara'ya gitti."

predictions = ner(text)

for prediction in predictions:
    print(prediction)
### Loading the Model Directly

```python
import torch
from transformers import AutoModelForTokenClassification, AutoTokenizer

model_id = "BUCOLIN/berturk-milliyetner-ner"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForTokenClassification.from_pretrained(model_id)

text = "Mustafa Kemal Ankara'ya gitti."

inputs = tokenizer(
    text,
    return_tensors="pt",
    truncation=True,
    max_length=256,
)

model.eval()

with torch.no_grad():
    outputs = model(**inputs)

prediction_ids = outputs.logits.argmax(dim=-1)[0].tolist()
tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0])

for token, prediction_id in zip(tokens, prediction_ids):
    label = model.config.id2label[prediction_id]
    print(f"{token:<20} {label}")