Downloads · 30 days
72
61% of all-time downloads
ispras-crawlers/newsxlm-markuplm-en-ae
newsxlm-markuplm-en-ae is a token classification model from ispras-crawlers. Use it when you need labels on individual words, such as names. The card lists the license as mit.
[](https://ispranproceedings.elpub.ru/jour/article/view/2164/1961) [](https://huggingface.co/spaces/pbedrin/newsxlm-demo)
Downloads · 30 days
72
61% of all-time downloads
All-time downloads
119
Public
Parameters
135M
539 MB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors539 MB · 99%
From the Hugging Face model README
This repository contains the fine-tuned MarkupLM baseline evaluated in the NewsXLM paper for news page HTML node classification. This version translates HTML node text into English with Argos Translate before feature extraction. Predictions contain the original text.
It labels HTML nodes as title, publication_date, text, author, tag, or none.
pip install -r requirements.txt
For non-English HTML, the required Argos translation model is downloaded and installed automatically on first use.
Pass the source language as lang="ru" to the processor or either pipeline.
The default is lang="en", which skips translation for English HTML. Argos doesn't support lang="auto".
If translation models are stored elsewhere, set ARGOS_PACKAGES_DIR before loading the processor.
Argos automatically uses CUDA when torch.cuda.is_available(), otherwise CPU.
Set ARGOS_DEVICE_TYPE=cpu to force CPU translation.
The Russian example places the author in tail text after an icon element.
html = '''
<html><body>
<nav><a>Главная</a><a>Новости</a></nav>
<main><article>
<h1>Учёные открыли новую планету за пределами Солнечной системы</h1>
<div class="author"><span class="icon-user"></span>Джон Смит</div>
<time>3 ноября 2026, 10:30</time>
<div class="body">
<p>Учёные сообщили об открытии во вторник.</p>
<p>Планета в два раза больше Земли.</p>
</div>
<div><strong>Теги</strong><a>Наука</a><a>Космос</a></div>
</article></main>
<footer>Все права защищены</footer>
</body></html>
'''
The extraction pipeline groups the results by attribute:
from transformers import pipeline
extractor = pipeline(
"extraction",
model="ispras-crawlers/newsxlm-markuplm-en-ae",
trust_remote_code=True,
)
extracted = extractor(html, lang="ru")
print(extracted)
Example result:
{
"title": ["Учёные открыли новую планету за пределами Солнечной системы"],
"publication_date": ["3 ноября 2026, 10:30"],
"text": ["Учёные сообщили об открытии во вторник.", "Планета в два раза больше Земли."],
"author": ["Джон Смит"],
"tag": ["Наука", "Космос"]
}
The node-classification pipeline returns a prediction for each HTML node.
from transformers import pipeline
node_classifier = pipeline(
"node-classification",
model="ispras-crawlers/newsxlm-markuplm-en-ae",
trust_remote_code=True,
)
nodes = node_classifier(html, lang="ru")
print(nodes)
Example result:
{
"/html/body/main/article/h1": {
"pred": "title",
"probs": {
"title": 0.98,
"publication_date": 0.0,
"text": 0.0,
"author": 0.0,
"tag": 0.0,
"none": 0.02
},
"text": "Учёные открыли новую планету за пределами Солнечной системы"
}
}
/following-sibling::text()[1]. Paths refer to the cleaned DOM.chunk_batch_size when creating the pipeline or calling it.import torch
from transformers import AutoModelForTokenClassification, AutoProcessor
repo_id = "ispras-crawlers/newsxlm-markuplm-en-ae"
processor = AutoProcessor.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForTokenClassification.from_pretrained(repo_id)
model.eval()
inputs = processor(html, lang="ru", return_tensors="pt")
with torch.inference_mode():
outputs = model(**inputs)
If you use NewsXLM in your research, please cite:
@article{newsxlm,
author = {Bedrin, P. A. and Varlamov, M. I. and Yatskov, A. K.},
title = {NewsXLM: A Multilingual Dataset and Model for Information Extraction from News Web Pages},
journal = {Proceedings of ISP RAS},
year = {2026},
volume = {38},
number = {2},
pages = {149--164},
doi = {10.15514/ISPRAS-2026-38(2)-10}
}