morisien-embed-v1.5

morisien-embed-v1.5

A sentence embedding model for Mauritian Creole (Kreol Morisien, mfe), fine-tuned from LaBSE.

This supersedes morisien-embed for Creole to English retrieval. That model is left in place: it is the checkpoint described in the preprint and in the MTEB task, and its weights are unchanged.

What changed

The first version was fine-tuned from multilingual-e5-base. On a benchmark hard enough to separate models it scores below untrained LaBSE, so its advantage did not hold outside the domain it was trained on. This version starts from LaBSE instead and is the first configuration measured that beats untrained LaBSE on that benchmark.

The gain is modest and the honest figures are below.

Results

Creole to English. The in-domain split is the held-out MorisienMT test set. The out-of-domain pool is FLORES+ devtest with distractors from the released xSIM++ augmentation, 996 queries over 45,029 passages, where each distractor is a gold passage with one meaning-critical token changed.

Three seeds of this recipe give 0.9655 (sd 0.0005) in-domain and 0.8597 (sd 0.0021) out of domain. The released checkpoint is seed 42, from its own release run, and its row above is measured on the released weights at revision 954c0137. Earlier versions of this card showed 0.9654 and 0.8616 there, which were a different seed-42 run's training-time scores. Run-to-run drift on an identical command reaches 0.0024 on the out-of-domain pool, which is worth knowing before reading small differences.

On three-seed means, the fine-tune is worth +0.026 in-domain and +0.015 out of domain against LaBSE. Against the first version's recipe it holds the in-domain score and adds 0.017 out of domain.

Other retrieval directions

Held-out MorisienMT test, nDCG@10, all three measured in one run in which morisien-embed reproduces its published scores exactly.

MTEB MorisienMTBitextMining

F1 on task revision 45f511e8. LaBSE and morisien-embed reproduce their published means of 0.848 and 0.925, so all three rows are on the same footing.

The four subsets are built over one set of 999 Creole sentences rather than four independent samples, so the mean is not an average of four independent measurements.

Matryoshka truncation

Creole to English, nDCG@10. Half the embedding costs about 0.005.

Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("Singaraj/morisien-embed-v1.5")
creole = model.encode(["Mo pe al bazar aster."])
english = model.encode(["I am going to the market now."])

No query or passage prefix is used at training or inference.

An fp16 ONNX export of the transformer is published at onnx/model.onnx. It outputs last_hidden_state, so pooling, the dense layer and normalisation still come from modules.json. Embeddings agree with the PyTorch weights to 1.7e-04, and in-domain Creole to English retrieval scores the same accuracy@1 of 0.9460.

A second export at browser/model.onnx carries the whole pipeline, 531 MB. CLS pooling, the dense layer with tanh and L2 normalisation are inside the graph, so it returns sentence_embedding and needs no pooling from the caller. It sits outside onnx/ on purpose: everything in that directory is expected to return last_hidden_state, and both the sentence-transformers backend and a transformers.js dtype lookup would pool a finished embedding a second time if they found this one there. Load it by path with onnxruntime, not through either of those.

There is deliberately no int8 export. LaBSE does not survive dynamic int8 quantisation here: accuracy@1 falls from 0.9460 to 0.7420, so a q8 file would quietly serve a much worse model.

The split that does work is the reverse of what saves the most bytes by instinct. Quantising the embedding table, which is 82% of the file, costs almost nothing; quantising the transformer matrix multiplications is what breaks the model. Measured on Creole to English retrieval against 0.9460 for the PyTorch weights: int8 everywhere 0.7420, int8 matrix multiplications only 0.8520, int8 embedding table only 0.9440. browser/model.onnx is the last of those with the remainder in fp16, and it holds in all three directions, scoring 0.9440, 0.9480 and 0.9379 against 0.9460, 0.9490 and 0.9389.

Training

LaBSE fine-tuned on 35,064 Creole to English and Creole to French pairs from MorisienMT and the Mauritian portion of Kreyol-MT. A first-stage model mines hard negatives with positive-aware filtering, five per anchor at a minimum rank of 10 and a relative margin of 0.05, which leaves roughly 24,100 pairs with a full negative set. That count varies between runs: four recorded ablation runs mining from the same first-stage checkpoint gave between 24,096 and 24,101, and the release run's own count was not recorded. The released checkpoint trains on those pairs with a cached multiple-negatives ranking loss at batch 128, wrapped in a Matryoshka loss over 768, 512, 256, 128 and 64 dimensions, for three epochs at learning rate 2e-5 on one T4.

Code, data construction and the evaluation harness are at LK-maker-007/morisien-embed.

What is known about its weaknesses

The margin over LaBSE is small. 0.015 nDCG@10 out of domain, against a measured run-to-run drift of 0.0024. It is consistent across three seeds with no overlap, and it is not large.

Most of what the model knows comes from LaBSE. LaBSE lists 110 languages and Mauritian Creole is not among them, but Haitian Creole is. Its starting point on this language is transfer from a related French-lexified creole. Fine-tuning is worth 0.174 out of domain when applied to multilingual-e5-base and 0.015 when applied to LaBSE, which is what that head start looks like.

Haitian Creole is close enough to confuse it. On FLORES+ triplets, injecting a same-meaning Haitian twin for every sentence drops Creole to English accuracy@1 substantially for every multilingual model tested. Mixed Mauritian and Haitian corpora will degrade retrieval.

The training corpus is largely a dictionary. 22,164 of the 35,064 pairs are a single Creole word, and the median Creole side is one word. Removing them makes the model worse, and so does adding 2,468 sentence-level pairs from google/smol, so the corpus appears to sit near a local optimum for this recipe rather than being straightforwardly improvable.

Both this model and LaBSE fail the same way. On the out-of-domain pool, distractors made by reversing a causal relation are 4.35% of the corpus and cause 49.7% of this model's errors and 46.8% of LaBSE's, while entity substitutions are 88.2% of the corpus and cause about a third of each. Neither model reads causal direction reliably.

On the metric the benchmark defines. xSIM++ scores with a margin-based similarity rather than plain cosine, and the error rates above use it: 0.2932 for this model against 0.3343 for untrained LaBSE, McNemar exact p = 0.00083 and a paired bootstrap 95% interval of [-0.0653, -0.0181] over 996 queries. Under plain cosine the same comparison is 0.2751 against 0.3072, so the advantage is slightly larger on the specified scoring than on the one the first version reported.

Sequence length is 256 tokens, LaBSE's default, against 512 for the first version. Every benchmark used here is single sentences, so nothing measured exercises the difference.

Loads on sentence-transformers 5.4 and newer. On 5.4.0 and 5.7.0 it gives identical embeddings, and on 5.7.0 it reproduces the Creole to English figures above exactly, 0.9658 nDCG@10 and 0.9460 accuracy@1. An earlier revision of this card said 6.0 was required. That stopped being true on 2026-09-06, when modules.json was changed to load Normalize from a path that 5.x has too.

Citation

The preprint covers both released versions. The DOI below is the concept DOI, so it always resolves to the newest revision.

@misc{morisien-embed,
  author    = {Singaraj B},
  title     = {morisien-embed: Text Embedding Models and Evaluation for Mauritian Creole (Kreol Morisien)},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.21877805},
  url       = {https://doi.org/10.5281/zenodo.21877805}
}