modern_bert_bg_base_uncased

Model Card for Model ID

ModernBERT model trained on Bulgarian literature, Web, and other datasets - uncased.

Model Details

124M parameter ModernBERT model trained on 29B (35B depending on tokenization) tokens for 3 epochs with Masked Language Modelling objective.

• Tokenizer vocabulary size is 50176.

• Model hidden dimension is 768.

• Feed-Forward dimension is 1152.

• Hidden layer count is 22.

• Developed by: Artificial Inteligence and Language Technologies Department at Institute of Information and Communication Technologies - Bulgarian Academy of Sciences.

• Funded by: The model is pretrained within the CLaDA-BG: National Interdisciplinary Research E-Infrastructure for Bulgarian Language and Cultural heritage - member of the pan-European research consortia CLARIN-ERIC & DARIAH-ERIC, funded by the Ministry of Education and Science of Bulgaria (support for the Bulgarian National Roadmap for Research Infrastructure). The training was performed at the supercomputer HEMUS at IICT-BAS, part of the RIs of the CoE on Informatics and ICT, financed by the OP SESG (2014–2020), and co-financed by the European Union through the ESIF.

• Model type: ModernBERT

• Language(s) (NLP): Bulgarian.

• License: MIT

Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

The model is intended to be used as a base model for fine-tuning tasks in NLP.

Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

>>> from transformers import (
>>>     PreTrainedTokenizerFast,
>>>     ModernBertForMaskedLM,
>>>     pipeline 
>>> )

>>> model = ModernBertForMaskedLM.from_pretrained('AIaLT-IICT/modern_bert_bg_base_uncased')
>>> tokenizer = PreTrainedTokenizerFast.from_pretrained('AIaLT-IICT/modern_bert_bg_base_uncased')

>>> fill_mask = pipeline(
>>>     "fill-mask",
>>>     model=model,
>>>     tokenizer=tokenizer
>>> )


>>> fill_mask("Заради 3 завода няма да [MASK] нито есенниците неподхранени, нито зърното да поскъпне заради тях.")

[{'score': 0.4933295249938965,
  'token': 17875,
  'token_str': 'останат',
  'sequence': 'заради 3 завода няма да останат нито есенниците неподхранени, нито зърното да поскъпне заради тях.'},
 {'score': 0.06887241452932358,
  'token': 12941,
  'token_str': 'има',
  'sequence': 'заради 3 завода няма да има нито есенниците неподхранени, нито зърното да поскъпне заради тях.'},
 {'score': 0.06286628544330597,
  'token': 13311,
  'token_str': 'бъдат',
  'sequence': 'заради 3 завода няма да бъдат нито есенниците неподхранени, нито зърното да поскъпне заради тях.'},
 {'score': 0.05207058787345886,
  'token': 26913,
  'token_str': 'оставим',
  'sequence': 'заради 3 завода няма да оставим нито есенниците неподхранени, нито зърното да поскъпне заради тях.'},
 {'score': 0.05052710324525833,
  'token': 12861,
  'token_str': 'са',
  'sequence': 'заради 3 завода няма да са нито есенниците неподхранени, нито зърното да поскъпне заради тях.'}]

Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->

The model is not trained on Next Sentence prediction so the [CLS] token embedding will not be useful out of the box. If you want to use the model for Sequence classification it is recommended to fine-tune it.

Recommendations

It is recommended to use the model for Token Classification and Sequence classification fine-tuning tasks. The model can be used within SentenceTransformers framework for producing embeddings.

Training Details

Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

Trained on 29B tokens consisting of deduplicated union of:

• uonlp/CulturaX
• MaCoCu-bg 2.0
• HPLT 2.0 Bulgarian (Cyrillic) cleaned
• Literature
• Wikipedia
• others

Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

Trained with Masked Language Modelling with 20% masks for 3 epochs with bf16 mixed precision, 512 tokens context and batch size of 256*512 tokens. After this training session, the context length was extended to 8,192 tokens through additional training with a batch size of 128 tokens and approximately 900K documents with longer contexts, filtered from the training data, totaling about 7B words.

Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

The model is evaluated on the Masked Language Modelling objective on test split with 20% random masked tokens. It achieves test loss of 0.99 and test accuracy of 77.58%

Citation

@inproceedings{simov-etal-2026-small,
  title = {Small Can Be Beautiful in LLMs for SSH: a Case for Bulgarian },
  author = {Simov, Kiril and Paev, Nikolay and Osenova, Petya and Valchev, Teodor and Marinov, Stefan},
  booktitle = {Proceedings of Shaping Multilingual, Multimodal AI for the Social Sciences and Humanities (LLMs4SSH) @ LREC 2026},
  month = {May},
  year = {2026},
  pages = {187--197},
  address = {Palma, Mallorca, Spain},
  publisher = {European Language Resources Association (ELRA)},
  editor = {Montejo-Raez, Arturo and Grisot, Cristina and Blochowiak, Joanna and Ljubešić, Nikola and Battaner, Elena and Rigau, German},
  doi = {10.63317/5n9bq7g3it2s},
  abstract = {In the paper we present a set of small LLM-based models for solving the basic NLP tasks for Bulgarian - POS tagging, Lemmatization, Dependency parsing, Named Entity Recognition, Named Entity Linking, Event Annotation, among others. In order to create fine-tuned models for these tasks, we first pre-train models using architectures like BERT, Modern-BERT, and T5 with different sizes, over Bulgarian data only. For each of the tasks we report our approach towards the fine-tuning, the results from the experiments and also the evaluation. Then we define a way to visualize the results over HTML documents which contain the analyzed texts. Our rationale are as follows: most, if not all SSH research scenarios, need a reliable processing chains that can be customized with respect to the specific needs. These scenarios also would need proper visualization for human observation. We aim to provide such a basic LLM-based toolkit.}
}

Model Card Authors

Nikolay Paev, Kiril Simov

Model Card Contact

[email protected]