turkish_focused-multilingual-e5-small

turkish_focused-multilingual-e5-small (V4 LoRA)

This is a domain adapted embedding model based on intfloat/multilingual-e5-small. It has been fine-tuned using Low-Rank Adaptation (LoRA) to significantly boost Turkish-language retrieval and cross-lingual alignment while strictly preserving the base model's English scientific and factoid retrieval capabilities.

🚀 Architecture & Training

To avoid catastrophic forgetting of pre-trained weights, this model utilizes Low-Rank Adaptation (LoRA) via PEFT. Only the attention matrices (query, key, value, dense) were adapted, keeping base representations stable.

• Base Model: intfloat/multilingual-e5-small
• LoRA Config: $r=16$, $\alpha=32$, dropout $= 0.05$
• Epochs: 1.5
• Batch Size: 160
• Learning Rate: 3e-5 (warmup: 281 steps)
• Hardware: Local NVIDIA GeForce RTX 4060 (8GB VRAM)

📚 Training Dataset Composition (300,000 Pairs)

The model was trained on a balanced ~300k-pair corpus designed to blend general QA, instruction-following, and specialized Turkish retrieval:

• English Retention Corpus (100,000 pairs):
  • rajpurkar/squad
  • yahma/alpaca-cleaned
  • databricks/databricks-dolly-15k
• Turkish General Corpus (150,000 pairs):
  • boun-tabi/squad_tr
  • merve/turkish_instructions
  • atasoglu/databricks-dolly-15k-tr
• Targeted Turkish Corpus (50,000 pairs):
  • 30,000 pairs: trmteb/turkish_embedding_model_training_data
  • 10,000 pairs: AhiskaAI/Ahiska-Turkish-Language-Dataset (passages paired with synthetic queries generated via gpt-5.6-luna and local LLM inference)
  • 10,000 pairs: NumanKaanKaratas/turkish-sentences (passages paired with synthetic queries generated via gpt-5.6-luna and local LLM inference)

📊 MTEB Benchmark Evaluation

Evaluated across 11 tasks comparing the merged LoRA model against the original intfloat/multilingual-e5-small base:

💻 Usage

The PEFT adapter matrices have been fused directly into the base weights (merge_and_unload()), allowing direct usage via sentence-transformers without needing peft:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("Yigit-Karaman/turkish_focused-multilingual-e5-small")

# E5 models require query/passage prefixes
query = "query: Bilgi erişimi modellerinde LoRA nasıl çalışır?"
doc = "passage: Düşük dereceli uyarlama (LoRA), temel ağırlıkları dondurup dikkat katmanlarına eğitilebilir matrisler ekleyerek çalışır."

embeddings = model.encode([query, doc], normalize_embeddings=True)
similarity = embeddings[0] @ embeddings[1]

print(f"Cosine Similarity: {similarity:.4f}")