morisien-embed

morisien-embed

A newer model exists. morisien-embed-v1.5 is fine-tuned from LaBSE instead and scores better out of domain, where this model does not beat untrained LaBSE. This checkpoint is unchanged and stays available: it is the one described in the preprint and registered in the MTEB task.

To our knowledge, the first dedicated text embedding model for Mauritian Creole (Kreol Morisien, mfe), the home language of roughly 90% of Mauritius (2022 census).

Fine-tuned from multilingual-e5-base on 35,064 Creole↔{English, French} pairs from MorisienMT and the Mauritian portion of Kreyòl-MT. On the held-out MorisienMT test split it beats every general multilingual embedding model we evaluated, including LaBSE, in all three measured retrieval directions (Creole→English, Creole→French, English→Creole).

Two corrections to earlier wording on this card. The corpus is not all publicly available Creole parallel text: google/smol carries 2,472 further gold pairs that it does not contain. And the margin over LaBSE holds in domain but not outside it, which is what the section below on FLORES+ now says.

Use it for semantic search, retrieval, RAG, bitext mining, or clustering over Kreol Morisien text.

Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("Singaraj/morisien-embed")

creole = ["Mo pe al bazar aster.", "Bann zanfan pe zwe dan lakour."]
english = ["I am going to the market now.", "The children are playing in the yard."]

similarity = model.similarity(model.encode(creole), model.encode(english))

Trained with Matryoshka loss, so embeddings can be truncated for faster search at a small, measured accuracy cost (ndcg@10 on the benchmark below: 0.9591 at 256 dims, 0.9531 at 128):

model = SentenceTransformer("Singaraj/morisien-embed", truncate_dim=256)

No prompt/prefix is required.

Results

Creole→English retrieval on the held-out MorisienMT test split (1,000 queries, leak-free against training data, enforced in the data pipeline by exact matching and by a punctuation-, case- and accent-insensitive check):

Creole→French, same protocol:

English→Creole, the reversed direction over the same pairs (999 queries retrieving 1,000 Creole passages, built with scripts/build_benchmark.py --reverse):

Generalization to an independent domain, FLORES+ mfe devtest (1,012 professionally translated sentences from Wikinews, Wikijunior and Wikivoyage, zero overlap with training data):

Both models sit at the ceiling of this benchmark. FLORES+ sentences are long and distinctive, so 1,012-way retrieval saturates, and this is evidence of no out-of-domain degradation rather than a margin over LaBSE.

That reading was still too generous to the pool. A character n-gram TF-IDF baseline with no neural model scores 0.8221 accuracy@1 on it, so retrieving the right passage among 1,012 needs little more than surface overlap. On a harder pool built from the same sentences with xSIM++ distractors, this model does not beat untrained LaBSE. See the preprint and morisien-embed-v1.5.

The contrastive stage was repeated with three random seeds over the same deterministically mined negative set; Creole→English test ndcg@10 across seeds: 0.9653 ± 0.0002 (accuracy@1 0.9433 ± 0.0006). The released checkpoint is seed 42, designated before results were seen.

Every number in the tables above is reproducible from the training repository (Matryoshka figures via scripts/evaluate.py --truncate-dim). The Haitian-proximity and case-sensitivity figures under Limitations come from an internal adversarial audit of the released checkpoint.

MTEB

The held-out MorisienMT test split is now a task in MTEB, MorisienMTBitextMining, the first Mauritian Creole task in the benchmark. This model is registered in MTEB and its scores are on the leaderboard.

Bitext-mining F1 across the four directional subsets:

This is bitext-mining F1, a different metric from the ndcg@10 retrieval numbers above. The model is trained on the MorisienMT corpus this split is drawn from, so MTEB records the result as in-domain (via training_datasets), not zero-shot.

Training

• Data: 35,064 unique, leak-free Creole↔{English, French} pairs, merged from MorisienMT (MIT) and Kreyòl-MT (mixed licenses; used for training only, not redistributed). Every MorisienMT dev/test sentence is checked against training by exact matching and by a punctuation-, case- and accent-insensitive check. Both are checks rather than removal steps: they drop 0 of the 69,525 raw rows, because the evaluation splits and the training sources are already disjoint. Verifiably absent is a stronger claim than removed, and it is the one the measurement supports.
• Recipe: hard-negative mining with positive-aware false-negative filtering (mine_hard_negatives: 5 negatives/anchor, range_min=10, relative_margin=0.05). The margin filter is strict: 24,100 of the 35,064 pairs survived with a full negative set, and the released checkpoint's contrastive stage trained on those 24,100 tuples (the stage-1 mining model itself was trained on all 35,064). Contrastive training uses CachedMultipleNegativesRankingLoss (batch 128, 767 in-batch negatives per anchor) wrapped in MatryoshkaLoss (dims 768/512/256/128/64). 3 epochs, lr 2e-5, warmup 10%, fp16, seed 42, single T4 GPU (~30 min contrastive + ~11 min mining).
• Base model: intfloat/multilingual-e5-base (278M parameters, MIT).

Limitations

• Not native-perfect. Accuracy@1 around 0.944 means roughly one query in eighteen ranks a wrong translation first. Strong, but below a human bilingual speaker.
• Register skew. The available Creole data over-represents religious text, politics, and literature; highly informal or technical registers are less covered.
• Small evaluation universe. Retrieval is measured over ~1,000-passage corpora, standard for bitext benchmarks, but absolute scores would be lower against web-scale corpora.
• One distribution family. MorisienMT and Kreyòl-MT overlap heavily, and the only fully independent evaluation domain for mfe (FLORES+) is saturated at this corpus size, so the margin over LaBSE is demonstrated in-domain only.
• Haitian Creole proximity. Like every multilingual embedder we tested, the model embeds Haitian Creole close to Mauritian Creole. Measured on the 1,012 aligned mfe/hat/eng FLORES+ devtest triplets (scripts/probe_haitian.py): with every same-meaning Haitian twin injected into the corpus, mfe→eng accuracy@1 drops from 1.00 to 0.68, and LaBSE resists this trap better (0.79). Asked instead to tell the two creoles apart (is the English sentence closer to its Mauritian or its Haitian translation?), the fine-tune picks Mauritian 709/1012 times vs LaBSE's 351/1012. Wrong-meaning Haitian text is never confused; mixed mfe/hat corpora will still degrade retrieval.
• Case sensitivity. ALL-CAPS text embeds measurably differently from its lower-case form (cosine ≈ 0.81 to the same sentence); caps-heavy text retrieves worse.
• English-only regression. Fine-tuning costs some pure-English semantic quality: STS-b test Spearman ≈ 0.79 vs the base model's ≈ 0.85. Use a general model for English-only workloads; this model is for Creole and Creole↔{English, French} work.
• Long inputs are truncated at the encoder's maximum sequence length; chunk long documents before embedding.
• Protocol note. During recipe development the held-out test score was printed at the end of each training run, so recipe selection had test visibility; an internal adversarial audit bounded the resulting optimism at ≤ ~0.01 ndcg. The 3-seed replication was run after the recipe was frozen. Leak filtering reserves the Creole side of every evaluation pair; English/French target texts are not reserved, and an audit found 1 of 999 benchmark passages also occurring in training as the translation of a different Creole sentence (dropping it moves ndcg@10 by less than 0.0001). The accent-insensitive half of the leak check was added after the released run; it verifiably leaves the training set byte-identical, since the sources were already disjoint at that level.
• Orthographic variation. Training data mixes pre- and post-2011 (Lortograf Kreol Morisien) spellings; performance on older orthography is untested.

Citation

If you use this model, please cite the accompanying report along with the data sources it builds on: MorisienMT (Dabre & Sukhoo, 2022) and Kreyòl-MT (Robinson et al., NAACL 2024).

@misc{morisien-embed,
  author    = {Singaraj B},
  title     = {morisien-embed: A Dedicated Text Embedding Model and Benchmark for Mauritian Creole (Kreol Morisien)},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.21877805},
  url       = {https://doi.org/10.5281/zenodo.21877805}
}