embed-modernbert-68m

<h1 align="center">embed-modernbert-68m</h1>

A Polish text embedding model designed as a general-purpose text encoder. The model is based on the ModernBERT architecture and encodes texts into dense vector representations that can be used for text understanding tasks, such as clustering, semantic similarity and few-shot classification. Its compact size makes it possible to run the model locally on limited hardware, including CPU-only environments and offers an efficient alternative to larger LLM-based approaches. This is especially useful for clustering and other large scale text processing tasks, where thousands or millions of documents need to be encoded efficiently.

Training

The model was trained in two stages, both based on knowledge distillation. In each stage a larger teacher embedding model was used to transfer semantic representation capabilities to a smaller student model. In this setting distillation means teaching the student to generate embeddings similar to those produced by the teacher for the same or semantically related texts.

In both stages the teacher model was Qwen3-Embedding-8B, selected for its strong performance in clustering, classification and text similarity tasks, as confirmed on MTEB benchmark. The student model was initialized from ettin-encoder-68m, an English encoder based on ModernBERT architecture.

1. The first stage used multilingual knowledge distillation, following the approach introduced in Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation. The goal was to transfer the embedding space of an already trained teacher model from English to Polish. Training was performed on Polish-English parallel data, where each document was available in both languages. The teacher model encoded the English version and the student was trained to produce a similar vector representation for the corresponding Polish text. As a result, the student model learned to place Polish and English texts with the same meaning close to each other in the embedding space. The model was trained for 5 epochs with a batch size of 128 texts (64 parallel pairs of Polish-English texts).

2. The second stage used more advanced distillation objective inspired by Jasper and Stella: distillation of SOTA embedding models. This stage was performed only on Polish texts and was designed to improve the quality of learned Polish representations. The objective combined three complementary losses:

   • cosine loss, which aligns student embeddings with teacher embeddings,
   • similarity loss, ensuring consistent similarity judgements between student and teacher,
   • relative similarity distillation loss, which teaches the student to preserve the teacher's relative ranking of similar texts.

   While the first stage was essential for transferring Polish text representation capabilities to the student model, the three similarity-based objectives used in the second stage resulted in a stronger alignment between the student and teacher embedding spaces. As in the first stage, the model was trained for 5 epochs with a batch size of 128, but using only Polish texts.

The training data contained approximately 20 million Polish-English document pairs, combining existing Polish-English parallel corpora with English embedding datasets translated into Polish.

Evaluation - clustering

To evaluate clustering quality a Polish clustering benchmark was created with eight datasets covering short texts, long documents, web pages, news articles, scientific abstracts and domain-specific banking content. The benchmark combines existing Polish tasks with datasets translated from English to Polish using an LLM. The included tasks are: 8TAGS (clustering headlines into 8 topic categories), Banking-Long (clustering longer finance and banking related articles into 14 topical sections), Banking-Short (clustering finance and banking headlines into the same 14 topics), CORTEX (clustering short texts by expressed emotion into 9 classes), Fineweb-Topics (clustering Polish web pages into 22 topic categories), Guardian-Articles (clustering translated Guardian articles into 29 editorial sections), PLSC (clustering scientific titles and abstracts by research field into 8 classes) and WikiNewsPL (clustering WikiNews articles into 9 news topics).

The evaluation results for the new models together with comparisons to popular Polish and multilingual embedding models are presented in Tables 1 and 2. For each model, documents were encoded into vector representations and clustered with K-Means algorithm. Table 1 reports clustering quality using ARI (Adjusted Rand Index), while Table 2 uses V-measure. Each clustering experiment was repeated five times and the results reported in below tables are average ARI/V-measure scores. As a reference point, the evaluation also includes a lexical baseline method based on TF-IDF representations reduced to dense vectors with Truncated SVD. This baseline represents texts using word-level lexical information rather than learned semantic embeddings.

<table> <thead> <tr> <th>Model</th> <th>8TAGS</th> <th>PLSC</th> <th>WikiNewsPL</th> <th>Banking-Short</th> <th>Banking-Long</th> <th>CORTEX</th> <th>Guardian-Articles</th> <th>Fineweb-Topics</th> <th>MEAN</th> </tr> </thead> <tbody> <tr> <td>Lexical baseline</td> <td>4.0</td> <td>18.6</td> <td>39.4</td> <td>6.9</td> <td>36.1</td> <td>1.2</td> <td>24.9</td> <td>18.8</td> <td>18.7</td> </tr> <tr> <td>sdadas/mmlw-roberta-base</td> <td>28.4</td> <td>20.8</td> <td>41.6</td> <td>15.7</td> <td>36.8</td> <td>19.1</td> <td>32.2</td> <td>23.0</td> <td>27.2</td> </tr> <tr> <td>sdadas/mmlw-roberta-large</td> <td>33.2</td> <td>20.1</td> <td>38.6</td> <td>19.5</td> <td>40.8</td> <td>19.4</td> <td>28.8</td> <td>22.0</td> <td>27.8</td> </tr> <tr> <td>intfloat/multilingual-e5-small</td> <td>24.8</td> <td>19.3</td> <td>33.7</td> <td>17.2</td> <td>34.5</td> <td>11.3</td> <td>29.7</td> <td>22.9</td> <td>24.2</td> </tr> <tr> <td>intfloat/multilingual-e5-base</td> <td>27.1</td> <td>19.2</td> <td>34.0</td> <td>17.1</td> <td>35.0</td> <td>14.9</td> <td>27.8</td> <td>21.9</td> <td>24.6</td> </tr> <tr> <td>intfloat/multilingual-e5-large</td> <td>31.6</td> <td>18.1</td> <td>35.3</td> <td>23.5</td> <td>37.3</td> <td>17.6</td> <td>30.4</td> <td>22.5</td> <td>27.0</td> </tr> <tr> <td>paraphrase-multilingual-MiniLM-L12-v2</td> <td>21.8</td> <td>15.4</td> <td>35.6</td> <td>10.2</td> <td>19.0</td> <td>12.0</td> <td>21.3</td> <td>19.3</td> <td>19.3</td> </tr> <tr> <td>BAAI/bge-m3</td> <td>29.3</td> <td>17.6</td> <td>37.9</td> <td>22.4</td> <td>38.2</td> <td>20.4</td> <td>27.2</td> <td>22.8</td> <td>27.0</td> </tr> <tr> <td>snowflake-arctic-embed-l-v2.0</td> <td>30.8</td> <td>16.3</td> <td>38.7</td> <td><strong>30.6</strong></td> <td><strong>41.3</strong></td> <td>11.1</td> <td>28.1</td> <td>20.5</td> <td>27.2</td> </tr> <tr> <td>embed-modernbert-68m</td> <td>44.0</td> <td><strong>22.8</strong></td> <td><strong>42.7</strong></td> <td>25.7</td> <td>38.9</td> <td>16.0</td> <td>39.7</td> <td>31.3</td> <td>32.7</td> </tr> <tr> <td>embed-modernbert-395m</td> <td><strong>46.7</strong></td> <td>22.5</td> <td>42.3</td> <td>26.8</td> <td>40.8</td> <td><strong>22.6</strong></td> <td><strong>40.7</strong></td> <td><strong>32.0</strong></td> <td><strong>34.3</strong></td> </tr> <tr> <td>embed-roberta-8k</td> <td>26.3</td> <td>17.4</td> <td>38.9</td> <td>23.2</td> <td>39.4</td> <td>21.0</td> <td>29.2</td> <td>25.5</td> <td>27.6</td> </tr> </tbody> </table>

Table 1. Comparison of clustering evaluation results using ARI metric.

<table> <thead> <tr> <th>Model</th> <th>8TAGS</th> <th>PLSC</th> <th>WikiNewsPL</th> <th>Banking-Short</th> <th>Banking-Long</th> <th>CORTEX</th> <th>Guardian-Articles</th> <th>Fineweb-Topics</th> <th>MEAN</th> </tr> </thead> <tbody> <tr> <td>Lexical baseline</td> <td>12.2</td> <td>26.5</td> <td>49.6</td> <td>27.3</td> <td>52.3</td> <td>4.2</td> <td>43.5</td> <td>36.3</td> <td>31.5</td> </tr> <tr> <td>sdadas/mmlw-roberta-base</td> <td>37.0</td> <td>29.6</td> <td>50.3</td> <td>28.2</td> <td>50.5</td> <td>24.3</td> <td>48.4</td> <td>42.4</td> <td>38.8</td> </tr> <tr> <td>sdadas/mmlw-roberta-large</td> <td>41.5</td> <td>29.7</td> <td>49.0</td> <td>32.2</td> <td>53.6</td> <td>27.3</td> <td>47.7</td> <td>41.5</td> <td>40.3</td> </tr> <tr> <td>intfloat/multilingual-e5-small</td> <td>33.4</td> <td>28.4</td> <td>43.1</td> <td>30.8</td> <td>48.0</td> <td>17.6</td> <td>45.6</td> <td>40.8</td> <td>36.0</td> </tr> <tr> <td>intfloat/multilingual-e5-base</td> <td>35.7</td> <td>28.3</td> <td>43.3</td> <td>30.7</td> <td>48.9</td> <td>22.7</td> <td>45.8</td> <td>39.4</td> <td>36.9</td> </tr> <tr> <td>intfloat/multilingual-e5-large</td> <td>39.0</td> <td>27.1</td> <td>43.9</td> <td>38.1</td> <td>51.6</td> <td>25.8</td> <td>47.3</td> <td>41.4</td> <td>39.3</td> </tr> <tr> <td>paraphrase-multilingual-MiniLM-L12-v2</td> <td>29.0</td> <td>25.3</td> <td>43.0</td> <td>21.1</td> <td>32.6</td> <td>18.2</td> <td>37.0</td> <td>33.9</td> <td>30.0</td> </tr> <tr> <td>BAAI/bge-m3</td> <td>36.6</td> <td>27.0</td> <td>46.2</td> <td>35.7</td> <td>52.2</td> <td>28.0</td> <td>46.3</td> <td>40.4</td> <td>39.1</td> </tr> <tr> <td>snowflake-arctic-embed-l-v2.0</td> <td>38.0</td> <td>25.7</td> <td>45.2</td> <td><strong>44.5</strong></td> <td><strong>55.0</strong></td> <td>17.4</td> <td>47.1</td> <td>38.2</td> <td>38.9</td> </tr> <tr> <td>embed-modernbert-68m</td> <td>48.0</td> <td>33.4</td> <td><strong>53.6</strong></td> <td>39.0</td> <td>52.8</td> <td>22.4</td> <td>55.1</td> <td>50.7</td> <td>44.4</td> </tr> <tr> <td>embed-modernbert-395m</td> <td><strong>52.6</strong></td> <td><strong>33.9</strong></td> <td>53.3</td> <td>40.2</td> <td>54.3</td> <td><strong>30.0</strong></td> <td><strong>55.7</strong></td> <td><strong>51.1</strong></td> <td><strong>46.4</strong></td> </tr> <tr> <td>embed-roberta-8k</td> <td>34.7</td> <td>28.1</td> <td>48.4</td> <td>37.1</td> <td>53.6</td> <td>28.0</td> <td>46.4</td> <td>44.4</td> <td>40.1</td> </tr> </tbody> </table>

Table 2. Comparison of clustering evaluation results using V-measure.

The results are consistent across ARI and V-measure with both metrics leading to a similar ranking of models. The embed-modernbert models achieve the highest scores in both evaluations. Even the smaller 68M variant outperforms other several larger Polish and multilingual encoders. The embed-roberta-8k model also performs well, although its scores are a few percentage points lower than ModernBERT-based models. Increasing the size of the embed-modernbert model provides a modest average gain of about 2 percentage points across the eight tasks for both metric, although the improvement is larger on some individual datasets.

Evaluation - few-shot classification

The models were evaluated in few-shot classification using the SetFit framework. SetFit is designed for efficient text classification with a very small number of labeled examples and consists of two steps: contrastive fine-tuning of the sentence encoder followed by training a classification head.

The evaluation covered four banking and finance classification tasks: FPB (sentiment classification of financial news into negative, neutral and positive classes), Banking77 (customer intent classification for banking related queries into 77 intent categories), Banking-Short (topic classification of finance and banking headlines into 14 topics), Banking-Long (the same topic classification task as in Banking-Short, but using full article texts together with titles). Datasets FPB and Banking77 were machine translated into Polish.

For each task, the models were trained using only 2, 4 or 8 examples per class sampled from training split. The tests sets were kept separate from this sampling process. Each experiment was repeated three times with different training samples and the results reported in Table 3 are mean accuracy scores. In total 12 models were evaluated across 4 tasks.

<table> <thead> <tr> <th rowspan="3">Model / task</th> </tr> <tr> <th colspan="3">FPB</th> <th colspan="3">Banking77</th> <th colspan="3">Banking-Short</th> <th colspan="3">Banking-Long</th> </tr> <tr> <th>2</th> <th>4</th> <th>8</th> <th>2</th> <th>4</th> <th>8</th> <th>2</th> <th>4</th> <th>8</th> <th>2</th> <th>4</th> <th>8</th> </tr> </thead> <tbody> <tr> <td>paraphrase-MiniLM-L12-v2</td> <td>36.56</td> <td>38.59</td> <td>41.92</td> <td>34.76</td> <td>47.31</td> <td>56.16</td> <td>19.27</td> <td>22.89</td> <td>33.07</td> <td>25.32</td> <td>31.68</td> <td>43.63</td> </tr> <tr> <td>multilingual-e5-small</td> <td>41.06</td> <td>42.54</td> <td>55.88</td> <td>45.31</td> <td>57.48</td> <td>70.85</td> <td>37.93</td> <td>46.51</td> <td>54.71</td> <td>55.42</td> <td>58.33</td> <td>63.91</td> </tr> <tr> <td>multilingual-e5-base</td> <td>39.14</td> <td>42.82</td> <td>61.52</td> <td>49.75</td> <td>67.16</td> <td>77.43</td> <td>33.01</td> <td>40.70</td> <td>53.34</td> <td>52.29</td> <td>57.75</td> <td>64.39</td> </tr> <tr> <td>bge-m3</td> <td>42.54</td> <td>47.46</td> <td>62.82</td> <td>58.02</td> <td>72.79</td> <td>77.84</td> <td>38.19</td> <td>53.39</td> <td>62.13</td> <td><strong>61.39</strong></td> <td>63.03</td> <td>69.58</td> </tr> <tr> <td>snowflake-arctic-embed-l-v2.0</td> <td>39.69</td> <td>49.28</td> <td>64.98</td> <td>57.37</td> <td>73.28</td> <td>77.57</td> <td><strong>44.20</strong></td> <td>53.62</td> <td>59.07</td> <td>60.20</td> <td>63.47</td> <td>70.35</td> </tr> <tr> <td>pplx-embed-v1-0.6b</td> <td>33.82</td> <td>40.00</td> <td>58.42</td> <td>58.03</td> <td>71.78</td> <td>79.19</td> <td>38.32</td> <td>51.54</td> <td>55.15</td> <td>58.22</td> <td>61.23</td> <td>69.20</td> </tr> <tr> <td>Qwen3-Embedding-0.6B</td> <td>44.12</td> <td><strong>61.75</strong></td> <td>69.42</td> <td>46.57</td> <td>65.18</td> <td>72.58</td> <td>32.43</td> <td>40.35</td> <td>45.94</td> <td>32.89</td> <td>21.38</td> <td>13.27</td> </tr> <tr> <td>mmlw-roberta-base</td> <td>44.23</td> <td>50.69</td> <td>63.75</td> <td>58.25</td> <td>69.80</td> <td>76.99</td> <td>33.61</td> <td>48.99</td> <td>58.57</td> <td>54.59</td> <td>58.38</td> <td>66.05</td> </tr> <tr> <td>mmlw-roberta-large</td> <td>44.67</td> <td>52.20</td> <td>66.25</td> <td>58.60</td> <td>71.01</td> <td>76.00</td> <td>38.93</td> <td>54.63</td> <td>61.71</td> <td>56.83</td> <td>62.88</td> <td>68.48</td> </tr> <tr> <td>embed-modernbert-68m</td> <td><strong>46.08</strong></td> <td>51.75</td> <td>65.60</td> <td>48.93</td> <td>65.66</td> <td>74.32</td> <td>38.73</td> <td>56.69</td> <td>61.96</td> <td>58.16</td> <td>63.88</td> <td>71.87</td> </tr> <tr> <td>embed-modernbert-395m</td> <td>29.76</td> <td>49.59</td> <td>68.01</td> <td>56.78</td> <td><strong>74.71</strong></td> <td><strong>81.06</strong></td> <td>37.69</td> <td><strong>58.62</strong></td> <td>60.90</td> <td>55.25</td> <td>62.02</td> <td><strong>73.11</strong></td> </tr> <tr> <td>embed-roberta-8k</td> <td>45.46</td> <td>57.39</td> <td><strong>72.99</strong></td> <td><strong>61.01</strong></td> <td>74.31</td> <td>76.51</td> <td>41.81</td> <td>54.28</td> <td><strong>63.12</strong></td> <td>59.45</td> <td><strong>64.81</strong></td> <td>69.79</td> </tr> </tbody> </table>

Table 3. Comparison of few-shot classification results using 2, 4 and 8 training examples per class.

Across the few-shot classification experiments, the strongest overall performance was achieved by embed-modernbert-395m and embed-roberta-8k, each obtaining the highest average accuracy scores in 4 of the evaluated settings. The few cases where other models performed better occured in the most restrictive 2-shot and 4-shot setting. Some results show high variability between runs, especially on FPB. Since the task has only three sentiment classes, few-shot setups provide very small training examples in total, making the final fine-tuned model highly dependent on selected samples.

Usage

This is a general-purpose embedding model optimized for different types of tasks. The task is indicated by a prefix added at the beginning of the input text. The selected prefix has a significant effect on the resulting embedding vector and therefore should match intended use case:

• "[cls]: " for classification and clustering tasks,
• "[sts]: " for semantic similarity tasks,
• "[query]: " for search queries in information retrieval tasks,
• no prefix for documents and passages in information retrieval tasks.

For optimal performance, we recommend using sentence-transformers version 5.4.0 or later and initializing the model with dtype="float16" or dtype="bfloat16" together with attn_implementation="flash_attention_2".

from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim

from sklearn.cluster import KMeans


model = SentenceTransformer(
    "PKOBP/embed-modernbert-68m",
    device="cuda",
    model_kwargs={
        "dtype": "bfloat16",
        "attn_implementation": "flash_attention_2"
    }
)

# Clustering example
cls_prefix = "[cls]: "
documents = [
    cls_prefix + "Jak zastrzec kartę płatniczą w aplikacji mobilnej?",
    cls_prefix + "Ustawianie limitów transakcji kartowych dla płatności internetowych.",
    cls_prefix + "Warunki wcześniejszej spłaty kredytu hipotecznego.",
    cls_prefix + "Dokumenty wymagane przy składaniu wniosku o kredyt mieszkaniowy.",
    cls_prefix + "Zasady otwierania lokaty terminowej dla klientów indywidualnych.",
    cls_prefix + "Porównanie funduszy inwestycyjnych dostępnych w ofercie banku."
]
embeddings = model.encode(documents, convert_to_tensor=False)
cluster_ids = KMeans(n_clusters=3, random_state=42, n_init=10).fit_predict(embeddings)
print(cluster_ids)

# Semantic similarity example
sim_prefix = "[sts]: "
sentences = [
    sim_prefix + "Konto osobiste można otworzyć całkowicie online.",
    sim_prefix + "Proces zakładania rachunku jest dostępny w pełni zdalnie.",
    sim_prefix + "Przelewy zagraniczne są realizowane zgodnie z tabelą opłat i prowizji."
]
embeddings = model.encode(sentences, convert_to_tensor=True)
print(cos_sim(embeddings, embeddings))

# Retrieval example
query_prefix = "[query]: "
query = query_prefix + "Jak zwiększyć maksymalną kwotę pojedynczego przelewu?"
documents = [
    "Ustawienia limitów transakcyjnych są dostępne w bankowości internetowej i aplikacji mobilnej.",
    "Lokata terminowa może zostać odnowiona automatycznie po zakończeniu okresu umownego.",
    "Historia operacji na rachunku jest dostępna po zalogowaniu do serwisu."
]
query_emb = model.encode(query, convert_to_tensor=True)
doc_emb = model.encode(documents, convert_to_tensor=True)
best_match = cos_sim(query_emb, doc_emb).argmax().item()
print(documents[best_match])

Acknowledgements

This project is financed by the European Funds, registered under the number FENG.01.01-IP.01-A028/23-00. It focuses on "Building innovative large language models and a service platform for serving multi-task models within the Bank." The outcomes of this project are the result of a collaboration between the AI Lab at the National Information Processing Institute (Ośrodek Przetwarzania Informacji Państwowy Instytut Badawczy) and the AI Team at PKO Bank Polski.

Funding Amount by the European Funds: 9,2 mln PLN