berturk-128k-turkish-news-classification

BERTurk-128k Turkish News Classification (9-class)

Fine-tuned dbmdz/bert-base-turkish-128k-cased (BERTurk, 128k cased vocabulary) for single-label, multiclass topic classification of Turkish news text into 9 categories.

The model was trained on the TRNEWS-2025 dataset (a large, real-world, and highly imbalanced Turkish news corpus) with class-balanced cross-entropy loss. On a held-out, stratified test split of 40,063 documents it reaches 95.95% accuracy, 0.9602 weighted-F1, and 0.9357 macro-F1.

• Developed by: thealper2
• Model type: BertForSequenceClassification (encoder-only Transformer)
• Language: Turkish (tr)
• Finetuned from: dbmdz/bert-base-turkish-128k-cased
• License: MIT (inherited from the BERTurk base model; see Licensing for dataset terms)
• Task: Multiclass text classification (topic / news-category)

---

Labels

The classifier outputs one of 9 mutually exclusive news categories. Labels are stored in the model config (id2label / label2id) in their original Turkish surface forms; English glosses are provided below for convenience.

Because id2label is baked into the config, pipeline(...) and model.config.id2label return the human-readable Turkish label directly — no external mapping file is required at inference time.

---

Quick start

With pipeline

from transformers import pipeline

clf = pipeline(
    "text-classification",
    model="thealper2/berturk-128k-turkish-news-classification",
    truncation=True,
    max_length=512,
)

clf("Milli takım, hazırlık maçında sahadan galibiyetle ayrıldı.")
# [{'label': 'Spor', 'score': 0.99...}]

# Full probability distribution over all 9 classes:
clf("Borsa İstanbul günü yükselişle kapattı, faiz beklentileri fiyatlandı.",
    top_k=None)

With AutoModel (manual, batched)

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_id = "thealper2/berturk-128k-turkish-news-classification"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id).eval()

texts = [
    "Yeni sürücüsüz araç teknolojisi trafik güvenliğini artırmayı hedefliyor.",
    "Antalya'da turizm sezonu erken açıldı, oteller yüksek doluluğa ulaştı.",
]

enc = tokenizer(texts, padding=True, truncation=True,
                max_length=512, return_tensors="pt")

with torch.no_grad():
    logits = model(**enc).logits
    probs = logits.softmax(dim=-1)

pred_ids = probs.argmax(dim=-1).tolist()
for text, pid, p in zip(texts, pred_ids, probs):
    print(f"{model.config.id2label[pid]:16s} ({p[pid]:.3f})  <- {text[:50]}")

Tip: the model was trained on article bodies (Haber Gövdesi) rather than headlines, so it performs best on full paragraphs/articles. Very short headlines may be classified less reliably. Always pass truncation=True, max_length=512.

---

Intended uses & limitations

Intended uses

• Automatic topic tagging / routing of Turkish news articles.
• Editorial content organization, feed categorization, and dataset labeling assistance.
• A strong Turkish-language baseline / backbone for downstream topic classification.

Out-of-scope / limitations

• Domain shift: trained on news text. Performance will degrade on other genres (tweets, product reviews, legal/medical documents, conversational text).
• Single-label only: the head is a softmax classifier — it forces exactly one category even for articles that legitimately span multiple topics (e.g. sports-economy). It cannot emit multiple labels.
• Label imbalance: the corpus is dominated by Magazin (~39% of the data). Even though class-balanced loss was used, the tail classes (Çevre, Turizm, Bilim-Teknoloji, Kültür-Sanat) show lower precision and are more prone to confusion (see the confusion analysis).
• No factuality/sentiment signal: the model predicts topic, not veracity, sentiment, or stance.
• Potential bias: the model reflects the editorial and demographic biases of the source Turkish news outlets in TRNEWS-2025.

---

Training data

• Dataset: TRNEWS-2025: A Multiclass Turkish News Text Dataset (IEEE DataPort, DOI 10.21227/vasv-dj22, Sengul Bayrak, 2025). Access requires an IEEE DataPort subscription.
• Input column: Haber Gövdesi (news body). Label column: Sınıf (class). Auxiliary columns (ID, Erişim Tarihi, URL) were dropped.

Preprocessing

Applied before tokenization (dataset-agnostic cleaning pipeline):

• HTML unescaping + tag stripping.
• URL stripping (http(s)://…, www.…).
• Whitespace normalization.
• Casing preserved (lowercase: false) — the base model is cased; no Turkish İ/ı folding.
• Exact-duplicate removal on the cleaned text.
• Empty/too-short rows removed (min_chars: 1).

Splits

Stratified train/validation/test split (seed=42), 90% / 5% / 5%:

Class distribution (full corpus, approximate)

The dataset is heavily imbalanced (≈20:1 between the largest and smallest class), which motivated class-weighted training.

(Counts are inferred from the stratified test-split support × 20; they are accurate to within rounding.)

---

Training procedure

Fine-tuned end-to-end with the HuggingFace Trainer (a custom WeightedTrainer subclass applying per-class weights to the cross-entropy loss).

Class-balanced loss

Cross-entropy is weighted with the scikit-learn "balanced" scheme, computed on the training split:

weight_c = N_train / (num_classes * count_c)

This yields weights ranging from ≈ 0.28 for the majority class (Magazin) to ≈ 5.7 for the minority class (Çevre), pulling macro-F1 up substantially relative to unweighted training.

Hyperparameters

Compute

• Hardware: single NVIDIA RTX 5060 Ti (Blackwell, sm_120, 8 GB VRAM).
• Frameworks: PyTorch (CUDA 12.8 wheel) · Transformers 4.57.6.
• 8 GB VRAM was made feasible through bf16, SDPA attention, dynamic padding, and length grouping (fixed 512-token padding was avoided).

Convergence & early stopping

Because the training set is large (~721k examples), the model converged within a fraction of the first epoch. Validation f1_macro peaked at step 3000 and did not improve over the next two evaluations, so EarlyStoppingCallback halted training at step 4000 and restored the best (step-3000) checkpoint. Selected validation metrics:

Note: the low epoch count is expected, not a bug — a single epoch over ~721k documents is a large number of optimization steps. Training longer offered no macro-F1 gain and risked overfitting the majority class.

---

Evaluation

All metrics below are on the held-out test split (40,063 documents), which was never seen during training or model selection.

Overall

Per-class report

Confusion analysis

Rows = true label, columns = predicted label (order matches the label table above).

Key takeaways:

• High-support, well-separated classes (Magazin, Spor, Siyaset, Finans-Ekonomi) exceed 0.96 F1.
• The main error mode is precision loss on the minority classes Kültür-Sanat (0.82) and Bilim-Teknoloji (0.84): a slice of Magazin articles leaks into Kültür-Sanat (279 cases), and several Siyaset articles are pulled into Bilim-Teknoloji. These are semantically adjacent topics in Turkish news.
• Class weighting keeps recall high across every class (min. recall 0.92), which is often the desired trade-off for tail categories.

Reproducing evaluation

The exact metrics, full classification_report, and confusion matrix are shipped in test_report.json alongside these weights.

---

Model architecture

---

Licensing

• Model weights: released under the MIT License, inherited from the base model dbmdz/bert-base-turkish-128k-cased.
• Training data (TRNEWS-2025): distributed via IEEE DataPort and requires a subscription to access. This repository ships only the fine-tuned weights and evaluation artifacts — no raw dataset text is included or redistributed. If you use the data itself, you must comply with IEEE DataPort's terms and cite the dataset (below).

---

Citation

If you use this model, please cite both the underlying dataset and the base model.

Dataset (TRNEWS-2025):

@data{vasv-dj22-25,
  doi       = {10.21227/vasv-dj22},
  url       = {https://dx.doi.org/10.21227/vasv-dj22},
  author    = {Sengul Bayrak},
  publisher = {IEEE Dataport},
  title     = {TRNEWS-2025: A Multiclass Turkish News Text Dataset},
  year      = {2025}
}

Base model (BERTurk):

@software{stefan_schweter_2020_3770924,
  author    = {Stefan Schweter},
  title     = {BERTurk - BERT models for Turkish},
  month     = apr,
  year      = 2020,
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.3770924},
  url       = {https://doi.org/10.5281/zenodo.3770924}
}

This fine-tuned model:

@misc{berturk_128k_turkish_news_classification,
  author    = {thealper2},
  title     = {BERTurk-128k Turkish News Classification (9-class)},
  year      = {2025},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/thealper2/berturk-128k-turkish-news-classification}}
}

---

Acknowledgements

• Sengul Bayrak (Istanbul Sabahattin Zaim University) for the TRNEWS-2025 dataset.
• Stefan Schweter / dbmdz for the BERTurk base models.
• The Hugging Face transformers, datasets, and accelerate teams.