embeddingmagibu-200m

embeddingmagibu-200m

embeddingmagibu-200m is a Turkish-focused, multilingual sentence embedding model developed through cross-lingual tokenizer surgery, teacher-model cloning, and offline embedding distillation.

This model is released as part of the artifacts for the paper:

Adapting Multilingual Embedding Models to Turkish via Cross-Lingual Tokenizer Surgery and Offline Distillation
M. Ali Bayram, Banu Diri, Savaş Yıldırım
arXiv: https://arxiv.org/abs/2605.29992

The model produces 768-dimensional L2-normalized sentence embeddings and supports an 8,192-token context window, making it suitable for Turkish semantic search, retrieval-augmented generation, long-document representation, clustering, classification, and semantic textual similarity tasks.

Although the model is optimized for Turkish, it was adapted from a multilingual teacher and trained with a multilingual distillation setup, making it useful for Turkish-centered multilingual NLP scenarios.

Key Features

• Turkish-focused sentence embedding model
• Multilingual support through a 40-language adaptation setup
• 8,192-token context length
• 768-dimensional normalized embeddings
• Approximately 200M parameters
• SentenceTransformers-compatible architecture
• Trained through offline embedding distillation
• Built with a Turkish-optimized multilingual tokenizer
• Designed for semantic search, RAG, clustering, classification, and STS

Associated Paper

This model is one of the released artifacts of the following paper:

Adapting Multilingual Embedding Models to Turkish via Cross-Lingual Tokenizer Surgery and Offline Distillation
arXiv: https://arxiv.org/abs/2605.29992

The paper introduces an efficient adaptation pipeline for building Turkish-focused embedding models without expensive full pretraining. The pipeline consists of:

1. Constructing a Turkish-optimized multilingual tokenizer
2. Cloning the teacher embedding model while preserving transformer backbone weights
3. Distilling from precomputed teacher embeddings using an offline embedding dataset

Intended Use

This model is suitable for:

• Turkish semantic search
• Retrieval-augmented generation
• Long-context retrieval
• Semantic textual similarity
• Sentence and document embedding
• Clustering
• Classification using embeddings
• Multilingual retrieval with Turkish-centered applications
• Low-resource and morphologically rich language NLP research

Model Performance

1. Detailed TR-MTEB Results

The model was evaluated on Turkish embedding benchmark tasks. The table below summarizes the detailed task-level results.

2. Version Comparison: embeddingmagibu-200m vs embeddingmagibu-152m

The table below compares embeddingmagibu-200m with the previous embeddingmagibu-152m version on shared evaluation tasks.

Model Architecture

This model uses the SentenceTransformers format with the following pipeline. The max_seq_length value is set to 8192.

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: Gemma3TextModel
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Dense({'in_features': 768, 'out_features': 3072, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity'})
  (3): Dense({'in_features': 3072, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity'})
  (4): Normalize()
)

Training and Adaptation Pipeline

The model was not trained from scratch. Instead, it was developed using an efficient three-stage adaptation pipeline.

1. Tokenizer Surgery

A Turkish-optimized multilingual tokenizer was constructed with a vocabulary size of 131,072 tokens.

The goal of this step was to improve Turkish tokenization efficiency while keeping multilingual capacity. This was done by pruning redundant tokens from the teacher vocabulary and incorporating multilingual tokens using frequency analysis over a balanced 40-language corpus.

2. Teacher Model Cloning

Instead of randomly initializing the full student model, the transformer backbone weights of the teacher model were preserved. A compatible embedding table was initialized for the new vocabulary through mean-composition token mapping.

This allows the student model to inherit the teacher model's representational capacity while adapting its vocabulary to Turkish and multilingual usage.

3. Offline Embedding Distillation

The student model was trained to approximate the teacher embedding space using precomputed teacher vectors.

This avoids online teacher inference during training, making the process faster, cheaper, and easier to reproduce. The distillation objective uses embedding-space similarity, such as cosine similarity, between student outputs and stored teacher embeddings.

The associated precomputed embedding dataset is available here:

• https://huggingface.co/datasets/alibayram/wikipedia-40-langs-with-embeddings

Evaluation

STSbTR Results

The model was evaluated on STSbTR, a Turkish semantic textual similarity benchmark.

These results show that embeddingmagibu-200m performs strongly on Turkish semantic textual similarity and improves over the teacher model on this benchmark.

TR-MTEB Leaderboard Comparison

Usage

Installation

pip install -U sentence-transformers

Basic Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer(
    "alibayram/embeddingmagibu-200m",
    trust_remote_code=True
)

sentences = [
    "Bugün hava çok güzel.",
    "Dışarısı güneşli.",
    "Uzun bağlam gerektiren çok detaylı bir hukuki veya teknik metin..."
]

embeddings = model.encode(sentences, normalize_embeddings=True)

print(embeddings.shape)  # (3, 768)

Similarity Computation

similarities = embeddings @ embeddings.T
print(similarities)

When embeddings are normalized, dot product is equivalent to cosine similarity.

Query and Document Encoding

The model supports query/document-style encoding through SentenceTransformers.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer(
    "alibayram/embeddingmagibu-200m",
    trust_remote_code=True
)

query = "Yapay zeka modellerinde distillation nedir?"

documents = [
    "Distillation, büyük bir öğretmen modelin bilgisinin daha küçük bir öğrenci modele aktarılmasıdır.",
    "Yapay zeka günümüzde çok popüler.",
]

query_embedding = model.encode_query(query)
document_embeddings = model.encode_document(documents)

scores = model.similarity(query_embedding, document_embeddings)
print(scores)

Limitations

• Although the model supports an 8,192-token context window, very long inputs may require more GPU memory.
• For long documents, chunking may still be useful depending on the downstream retrieval setup.
• The model is optimized for Turkish-centered use cases, but multilingual behavior may vary across languages.
• The quality of the distilled model depends on the teacher model and the multilingual distillation corpus.
• Benchmark results may vary depending on evaluation settings, pooling behavior, precision, and prompt/query formatting.

Related Resources

Paper:

• https://arxiv.org/abs/2605.29992

Dataset:

• https://huggingface.co/datasets/alibayram/wikipedia-40-langs-with-embeddings

Project page:

• https://huggingface.co/spaces/magibu/embeddingmagibu-200m

GitHub repository:

• https://github.com/malibayram/embedding-trainer

Citation

If you use this model in academic work, please cite the associated paper:

@article{bayram2026embeddingmagibu,
  title={Adapting Multilingual Embedding Models to Turkish via Cross-Lingual Tokenizer Surgery and Offline Distillation},
  author={Bayram, M. Ali and Diri, Banu and Yıldırım, Savaş},
  journal={arXiv preprint arXiv:2605.29992},
  year={2026},
  url={https://arxiv.org/abs/2605.29992}
}

You may also cite the model repository:

@misc{embeddingmagibu_200m_2026,
  title={embeddingmagibu-200m: Long-Context Turkish Sentence Embeddings},
  author={Bayram, M. Ali},
  year={2026},
  url={https://huggingface.co/alibayram/embeddingmagibu-200m}
}

Model Card Authors and Contact

• M. Ali Bayram
• Hugging Face: https://huggingface.co/alibayram
• GitHub: https://github.com/malibayram