LT-ABS-extractive-xlm-roberta

Hierarchical XLM-RoBERTa for Lithuanian Extractive Summarisation

Model Information

Model Name: Hierarchical-XLM-RoBERTa Extractive Summariser

Base Model: FacebookAI/xlm-roberta-base

Architecture: Hierarchical sentence classifier (XLM-RoBERTa + positional embeddings + 2-layer Transformer + linear head)

Task: Extractive summarisation of Lithuanian texts

The model stacks four components on top of XLM-RoBERTa-base:

At inference time, sentences are selected using Maximal Marginal Relevance (MMR) with a token budget, balancing relevance and diversity.

How to Get Started with the Model

Use this model for Lithuanian extractive summarisation with Hugging Face transformers and safetensors libraries.

Environment Setup

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
pip install transformers datasets safetensors evaluate pandas stanza bert-score

Download input preprocessing script (preprocessing.py) from this repository.

Running the models

import stanza
from transformers import AutoTokenizer, AutoModel
from preprocessing import full_document_processing

# Sentence splitting (Stanza Lithuanian pipeline)
nlp  = stanza.Pipeline(lang="lt", processors="tokenize", tokenize_no_ssplit=True, use_gpu=True)
text = "Your article text here..."
doc  = nlp(text)
sentences = full_document_processing(text, nlp, article=True)

# Load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained("xlm-roberta-base")
model = AutoModel.from_pretrained(
    "VytautoDidziojoUniversitetas/LT-ABS-extractive-xlm-roberta",
    trust_remote_code=True,
)
model.eval()

summary_sentences = model.summarize(sentences, tokenizer)
print(" ".join(summary_sentences))

Uses

• Extractive summary generation from Lithuanian texts
• Applications: Law, Healthcare, Information Technology, and News topics

Training Details

Fine-tuned on a VytautoDidziojoUniversitetas/LT_Summarisation_Corpus covering four domains:

• IT — information technology articles
• Law — legal documents
• Medicine — medical texts
• News — news articles

Training Configuration

training:
  epochs: 6
  gradient_accumulation_steps: 8
  learning_rate: 2e-5
  warmup_ratio: 0.1
  optimizer: AdamW

Environment: Hugging Face Transformers
Hardware: 1× NVIDIA RTX A6000 ADA

Evaluation

Evaluated on the test split of VytautoDidziojoUniversitetas/LT_Summarisation_Corpus. Reference summaries are extractive (summary_extract column).

Citation

Please cite the model as:

Vytautas Magnus University and Vilnius University. 2026. Abstract Corpora for Artificial Intelligence, Extractive model. Hugging Face. https://huggingface.co/VytautoDidziojoUniversitetas/LT-ABS-extractive-xlm-roberta

BibTeX:

@misc{lt_acai_ext_2026,
  author    = {{Vytautas Magnus University} and {Vilnius University}},
  title     = {Lithuanian Abstract Corpora Extractive Model},
  year      = {2026},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/VytautoDidziojoUniversitetas/LT-ABS-extractive-xlm-roberta},
  note      = {Developed under the New Generation Lithuania plan, project 02-101-K-0001.}
}

Acknowledgement

Funded by the European Union — NextGenerationEU. Naujos kartos Lietuva.

This project is funded by the Recovery and Resilience Facility under the Lithuanian "Naujos kartos Lietuva" plan and by the State Budget of the Republic of Lithuania.

Project: Santraukų tekstynai dirbtiniam intelektui — Project No. 02-101-K-0001.