multilingual-IPTC-news-topic-classifier

Multilingual IPTC Media Topic Classifier

News topic classification model based on xlm-roberta-large and fine-tuned on a news corpus in 4 languages (Croatian, Slovenian, Catalan and Greek), annotated with the top-level IPTC Media Topic NewsCodes labels. The development and evaluation of the model is described in the paper LLM Teacher-Student Framework for Text Classification With No Manually Annotated Data: A Case Study in IPTC News Topic Classification (Kuzman and Ljubešić, 2025).

The model can be used for classification into topic labels from the IPTC NewsCodes schema and can be applied to any news text in a language, supported by the xlm-roberta-large.

Based on a manually-annotated test set (in Croatian, Slovenian, Catalan and Greek), the model achieves macro-F1 score of 0.746, micro-F1 score of 0.734, and accuracy of 0.734, and outperforms the GPT-4o model (version gpt-4o-2024-05-13) used in a zero-shot setting. If we use only labels that are predicted with a confidence score equal or higher than 0.90, the model achieves micro-F1 and macro-F1 of 0.80.

Intended use and limitations

For reliable results, the classifier should be applied to documents of sufficient length (the rule of thumb is at least 75 words).

Use example:

from transformers import pipeline

# Load a multi-class classification pipeline - if the model runs on CPU, comment out "device"
classifier = pipeline("text-classification", model="classla/multilingual-IPTC-news-topic-classifier", device=0, max_length=512, truncation=True)

# Example texts to classify
texts = [
    """Slovenian handball team makes it to Paris Olympics semifinal Lille, 8 August - Slovenia defeated Norway 33:28 in the Olympic men's handball tournament in Lille late on Wednesday to advance to the semifinal where they will face Denmark on Friday evening. This is the best result the team has so far achieved at the Olympic Games and one of the best performances in the history of Slovenia's team sports squads.""",
	"""Moment dog sparks house fire after chewing power bank An indoor monitoring camera shows the moment a dog unintentionally caused a house fire after chewing on a portable lithium-ion battery power bank. In the video released by Tulsa Fire Department in Oklahoma, two dogs and a cat can be seen in the living room before a spark started the fire that spread within minutes. Tulsa Fire Department public information officer Andy Little said the pets escaped through a dog door, and according to local media the family was also evacuated safely. "Had there not been a dog door, they very well could have passed away," he told CBS affiliate KOTV."""]

# Classify the texts
results = classifier(texts)

# Output the results
for result in results:
    print(result)

## Output
## {'label': 'sport', 'score': 0.9985264539718628}
## {'label': 'disaster, accident and emergency incident', 'score': 0.9957459568977356}

The code for massive corpora annotation with topic labels is available here.

IPTC Media Topic categories

The classifier uses the top-level of the IPTC Media Topic NewsCodes schema, consisting of 17 labels.

List of labels

labels_list=['education', 'human interest', 'society', 'sport', 'crime, law and justice',
'disaster, accident and emergency incident', 'arts, culture, entertainment and media', 'politics',
'economy, business and finance', 'lifestyle and leisure', 'science and technology',
'health', 'labour', 'religion', 'weather', 'environment', 'conflict, war and peace'],

labels_map={0: 'education', 1: 'human interest', 2: 'society', 3: 'sport', 4: 'crime, law and justice',
5: 'disaster, accident and emergency incident', 6: 'arts, culture, entertainment and media',
7: 'politics', 8: 'economy, business and finance', 9: 'lifestyle and leisure', 10: 'science and technology',
11: 'health', 12: 'labour', 13: 'religion', 14: 'weather', 15: 'environment', 16: 'conflict, war and peace'}

Description of labels

The descriptions of the labels are based on the descriptions provided in the IPTC Media Topic NewsCodes schema and enriched with information which specific subtopics belong to the top-level topics, based on the IPTC Media Topic label hierarchy.

Training data

The model was fine-tuned on the training split of the EMMediaTopic 1.0 dataset consisting of 15,000 news in four languages (Croatian, Slovenian, Catalan and Greek). The news texts were extracted from the MaCoCu-Genre web corpora based on the "News" genre label, predicted with the X-GENRE classifier. The training dataset was automatically annotated with the IPTC Media Topic labels by the GPT-4o model (yielding 0.72 micro-F1 and 0.73 macro-F1 on the test dataset).

The code for the development and evaluation of the model is available on this GitHub repository.

Label distribution in the training dataset:

Performance

The model was evaluated on a manually-annotated test set in four languages (Croatian, Slovenian, Catalan and Greek), consisting of 1,129 instances. The test set contains similar amounts of texts from the four languages and is more or less balanced across labels.

The model was shown to achieve micro-F1 score of 0.734, and macro-F1 score of 0.746. The results for the entire test set and per language:

Performance per label:

For downstream tasks, we advise you to use only labels that were predicted with confidence score higher or equal to 0.90 which further improves the performance.

When we remove instances predicted with lower confidence (229 instances - 20%), the model yields micro-F1 of 0.798 and macro-F1 of 0.80.

Fine-tuning hyperparameters

Fine-tuning was performed with simpletransformers. Beforehand, a brief hyperparameter optimization was performed and the presumed optimal hyperparameters are:

model_args = ClassificationArgs()

model_args ={
             "num_train_epochs": 5,
             "learning_rate": 8e-06,
             "train_batch_size": 32,
             "max_seq_length": 512,
             }   
      

Citation

If you use the model, please cite this paper and the model itself:

@ARTICLE{10900365,
  author={Kuzman, Taja and Ljubešić, Nikola},
  journal={IEEE Access}, 
  title={LLM Teacher-Student Framework for Text Classification With No Manually Annotated Data: A Case Study in IPTC News Topic Classification}, 
  year={2025},
  volume={},
  number={},
  pages={1-1},
  keywords={Data models;Annotations;Media;Manuals;Multilingual;Computational modeling;Training;Training data;Transformers;Text categorization;Multilingual text classification;IPTC;large language models;LLMs;news topic;topic classification;training data preparation;data annotation},
  doi={10.1109/ACCESS.2025.3544814}}

@misc{iptc_topic_model,
	author       = { Kuzman, Taja and Ljube{\v s}i{\'c}, Nikola },
	title        = {{ Multilingual IPTC News Topic Classifier}},
	year         = 2025,
	url          = { https://huggingface.co/classla/multilingual-IPTC-news-topic-classifier },
	doi          = { 10.57967/hf/4709 },
	publisher    = { Hugging Face }
}

Funding

This work was supported by the Slovenian Research and Innovation Agency research project Embeddings-based techniques for Media Monitoring Applications (L2-50070, co-funded by the Kliping d.o.o. agency).