mentee-embed-v3

mentee-embed-v3

A 41M-parameter trilingual text embedding model trained entirely from scratch — no pretrained backbone, no BERT, no RoBERTa. Random initialization only.

Developed by Team MenteE AI (menteeai.org) as part of an ongoing research effort to build competitive multilingual embeddings from the ground up for Arabic, English, and Urdu.

---

Key Facts

---

What "From Scratch" Means

Most embedding models fine-tune an existing pretrained encoder (BERT, RoBERTa, MPNet). mentee-embed-v3 does not. We:

1. Trained a custom BPE tokenizer on Arabic, English, and Urdu text
2. Initialized a 12-layer Transformer with random weights
3. Ran masked language modeling pretraining on 2.1M+ sentences
4. Applied two-round contrastive distillation with hard negative mining

No pretrained checkpoint was used at any stage.

---

Benchmark Results

All baselines evaluated under identical conditions on the same hardware.

Protocol A — In-batch Retrieval (pool ≈ 97 candidates)

Format: acc@1 / R@5 / MRR@10

Avg MRR@10 computed over 5 datasets: MIRACL-EN, MIRACL-AR, MIRACL-UR, xling EN-UR, MS-MARCO (val excluded from avg).

✅ mentee-embed-v3 beats all-MiniLM-L6-v2 (0.655 vs 0.449) on Protocol A avg MRR@10 — despite all-MiniLM being a pretrained model.

---

Protocol B — MIRACL Wikipedia Corpus Retrieval (full ranking, ~5K–15K passages)

Format: MRR@10 · R@5 · R@100

📌 Protocol B uses Wikipedia passages (MIRACL) — a challenging out-of-domain test for a model trained primarily on NLI and MS-MARCO data.

---

Protocol C — MS-MARCO Corpus Retrieval (10K passages, in-domain)

🔥 0.645 MRR@10 from a randomly initialized 41M model — trained on 2.1M triplets vs billions for the baselines.
Gap to MiniLM-multilingual: only 0.194.

---

Training Data

---

Training Pipeline

Stage 1 — MLM Pretraining
  Random init → masked language modeling on 2.1M sentences
  8,000 steps · batch=32 · vocab=50K BPE

Stage 2 — Distillation Round 1 (no hard negatives)
  Teacher: intfloat/multilingual-e5-base (768-dim)
  InfoNCE contrastive + relational distillation
  4,000 steps · batch=512 · temp=0.05

Stage 3 — Hard Negative Mining
  GPU-accelerated top-5 mining across full 2.1M corpus

Stage 4 — Distillation Round 2 (with hard negatives)
  Same objective + mined hard negatives per anchor
  10,000 steps · batch=512 · temp=0.05

---

Usage

# pip install torch transformers tokenizers huggingface_hub
from transformers import AutoModel, AutoTokenizer

tok   = AutoTokenizer.from_pretrained("MenteEAI/mentee-embed-v3", trust_remote_code=True)
model = AutoModel.from_pretrained("MenteEAI/mentee-embed-v3",   trust_remote_code=True)

sentences = [
    "Hello, how are you?",
    "مرحبا، كيف حالك؟",
    "ہیلو، آپ کیسے ہیں؟"
]

embeddings = model.encode(sentences, tokenizer=tok)
print(embeddings.shape)  # torch.Size([3, 384])

trust_remote_code=True is required — standard for custom-architecture models on HuggingFace. The code runs entirely on your machine.

Similarity search

from transformers import AutoModel, AutoTokenizer

tok   = AutoTokenizer.from_pretrained("MenteEAI/mentee-embed-v3", trust_remote_code=True)
model = AutoModel.from_pretrained("MenteEAI/mentee-embed-v3",   trust_remote_code=True)

query    = model.encode(["What is machine learning?"], tokenizer=tok)
passages = model.encode([
    "Machine learning is a subset of artificial intelligence.",
    "The weather today is sunny.",
    "تعلم الآلة هو فرع من فروع الذكاء الاصطناعي.",
], tokenizer=tok)

scores = query @ passages.T
print(scores)  # tensor([[0.81, 0.60, 0.79]])

---

Limitations

• Protocol B (Wikipedia retrieval) scores are lower than pretrained baselines — the model was not trained on Wikipedia-style passages
• Arabic and Urdu lag behind English due to less retrieval-specific training data (mMARCO Arabic/Urdu was unavailable in a compatible format)
• Vocabulary limited to 50K tokens trained on ~2.1M sentences — rare scripts and dialects may tokenize poorly
• Not evaluated on MTEB full suite yet

---

Citation

@misc{mentee-embed-v3-2026,
  title   = {How Far Can Multilingual Text Embeddings Be Trained From Scratch?
             A Compute-Efficient Study of Arabic, English, and Urdu},
  author  = {Shah, Syed Syab Ahmad and Sania, Shakeel and Hamza, Rustam and Mahboob, Iqbal},
  year    = {2026},
  doi     = {10.5281/zenodo.22117673},
  url     = {https://doi.org/10.5281/zenodo.22117673},
  note    = {MenteE AI. Apache-2.0 License}
}

---

About MenteE AI

Built by Syed Syab Ahmad Shah and Team MenteE AI. 🌐 menteeai.org · 📧 [email protected] Research paper: 10.5281/zenodo.22117673