SIGMA-BERT-short

SIGMA-BERT-short

Table of Contents

1. Model Summary
2. Model Details
3. Usage
4. Training
5. Training Data
6. Tokenizer
7. Limitations
8. Citation

Model Summary

SIGMA-BERT-short is a Slovenian encoder language model based on the ModernBERT architecture. It was pretrained from scratch using the masked language modeling (MLM) objective on a diverse collection of Slovenian text, including web, news, parliamentary proceedings, scientific publications, and user-generated content.

SIGMA-BERT-short shares the same architecture and parameter count as SIGMA-BERT-base but is optimized for sequences of up to 1,024 tokens, making it well suited for standard sentence-level and document-level language understanding tasks.

The model is intended to be fine-tuned for downstream Slovenian NLP tasks such as:

• Text classification (e.g. topic, sentiment, genre)
• Named entity recognition and other token classification tasks
• Extractive question answering
• Natural language inference / semantic textual similarity
• Dense retrieval and semantic search
• As a backbone for rerankers in retrieval-augmented generation (RAG) pipelines

SIGMA-BERT models are available in multiple context-length variants:

• SIGMA-BERT-short - 22 layers, 149M parameters, 1,024 token context
• SIGMA-BERT-base - 22 layers, 149M parameters, 8,192 token context
• SIGMA-BERT-large - 28 layers, 395M parameters, 8,192 token context

Model Details

SIGMA-BERT-short is built on the ModernBERT architecture and pretrained on Slovenian text with a Masked Language Modeling (MLM) objective.

Training

SIGMA-BERT-short was pretrained in two stages.

Stage 1

The first stage focused on learning general Slovenian language representations.

• Maximum sequence length: 1,024
• MLM masking probability: 15%
• Learning rate: 5e-4
• Optimizer: StableAdamW
• Weight decay: 0.1

Stage 2

The second stage continued pretraining from the Stage 1 checkpoint using a subset of longer, information-dense Slovenian documents.

The context window was kept unchanged at 1,024 tokens.

• Maximum sequence length: 1,024
• MLM masking probability: 15%
• Learning rate: 5e-5
• Optimizer: StableAdamW
• Weight decay: 0.1

Both stages used a warmup–plateau–cooldown learning rate schedule.

Training Data

The training corpus contains approximately 13.88 billion tokens of Slovenian text.

Stage 1 used a mixture of publicly available Slovenian corpora:

• FineWeb2 — Slovenian subset (slv_Latn)
• Janes 1.0 (Wiki, Blog, Forum, News) — Slovenian text subcorpora
• OpenScience — Slovenian academic and scientific texts
• siParl 4.0 — Slovenian parliamentary proceedings corpus
• MaCoCu-sl 2.0 — Slovenian web corpus

Stage 2 continued training using:

• OpenScience
• siParl 4.0

Tokenizer

SIGMA-BERT-short uses the same custom SentencePiece BPE tokenizer as the other SIGMA-BERT models.

The tokenizer was trained from scratch with a vocabulary size of 50,000 tokens and uses Metaspace pre-tokenization together with newline isolation and digit splitting to better preserve document structure and numeric information.

Usage

Load the model like any other transformers model:

from transformers import AutoTokenizer, AutoModel
 
tokenizer = AutoTokenizer.from_pretrained("LHRS-UM-FERI/SIGMA-BERT-short")
model = AutoModel.from_pretrained("LHRS-UM-FERI/SIGMA-BERT-short")

Masked language modeling

For MLM, load the model with AutoModelForMaskedLM and use the [MASK] token:

from transformers import AutoTokenizer, AutoModelForMaskedLM
 
model_id = "LHRS-UM-FERI/SIGMA-BERT-short"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id)
 
text = "Ljubljana je[MASK] mesto Slovenije."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)
 
masked_index = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id)
predicted_token_id = outputs.logits[0, masked_index].argmax(axis=-1)
print("Predicted token:", tokenizer.decode(predicted_token_id))

⚠️ Note on [MASK] spacing: SIGMA-BERT's tokenizer uses a Metaspace pre-tokenizer, which encodes the space before a word as part of that word's token (e.g. " mesto" → ▁mesto). Because of this, if you write [MASK] in a text string, it should be placed directly against the preceding word, with no space ("je[MASK] mesto", not "je [MASK] mesto"). Adding a literal space before [MASK] produces an extra, out-of-distribution ▁ token that the model never saw during training:

"je [MASK]"  → ['▁je', '▁', '[MASK]', ...]   ✗ stray '▁' token
"je[MASK]"   → ['▁je', '[MASK]', ...]        ✓ matches training

Limitations

• SIGMA-BERT-short is an encoder-only model trained with masked language modeling and cannot generate text.
• The model is optimized for sequences up to 1,024 tokens and is not intended for long-document understanding beyond this length.
• The model is primarily intended for Slovenian language tasks.
• Like other pretrained language models, it may reflect biases present in the training data.

Citation

If you use SIGMA-BERT-short in your work, please cite:

@misc{Borovic2026,
  title = {SIGMA-BERT-short},
  author = { Mladen Borovič and Marija Jovanova },
  year = {2026},
  url = {https://huggingface.co/LHRS-UM-FERI/SIGMA-BERT-short},
  doi = { 10.57967/hf/9823 },
  publisher = { Hugging Face }
}