Sungur-3x9B-Gemma

<img src="./Sungur-MoE.png"/>

Sungur-3x9B-T1

Sungur-3x9B-T1 is a Domain-Specialized Sparse Mixture of Experts (MoE) model designed for high-performance Turkish text generation. It is built upon the robust google/gemma-2-9b-it as its base foundation.

This model was constructed using Gemma-2-MoE, a custom library I developed to build Gemma 2 based MoE models. Unlike traditional merges that rely purely on random router initialization, this model uses Semantic Router Initialization followed by a targeted fine-tuning stage.

It combines the strengths of three 9B models—including my own Sungur-9B—into a single efficient inference engine.

Turkish Evaluation Benchmark Results (via malhajar17/lm-evaluation-harness_turkish)

Note on Benchmark Results: The evaluation results show a performance level that is generally consistent with the strongest individual expert models, rather than a significant leap forward often seen in fully trained MoE models. This is primarily because the model's core expert weights were frozen during the brief fine-tuning stage; only the router was trained to route tokens to the correct specialist. For future iterations, a full fine-tuning pass across all expert weights and the router is anticipated. This will allow the distinct expert models to learn to cooperate and complement each other, leading to a substantial gain in overall benchmark performance and cross-domain synergy.

Expert Composition (Top-2 Routing)

The model dynamically routes each token to the top 2 experts, combining logic, academic knowledge, and creative nuance.

Build & Training Methodology

1. Semantic Router Initialization

Using my Gemma-2-MoE builder, the initial router weights were calculated using Concept Algebra on the embedding space. $$\text{Expert Vector} = \text{Mean}(\text{Positive Prompts}) - 0.5 \times \text{Mean}(\text{Negative Prompts})$$

2. Router Fine-Tuning

After construction, the base model and expert weights were frozen. Only the router (gating) network was fine-tuned for 2 epochs using a synthetic dataset of 1,500 samples. This dataset contained prompt-expert pairs (e.g., a Python question paired with the Reasoner expert) to sharpen the router's decision-making and ensure prompts are sent to the correct domain specialist.

Build Configuration

This model was built using the following YAML configuration, which defines the experts and semantic routing logic:

<details><summary>See MoE config</summary>

base_model: google/gemma-2-9b-it
dtype: bfloat16
num_experts_per_tok: 2

experts:
  # EXPERT 0: THE REASONER (Logic & Code)
  - source_model: ytu-ce-cosmos/Turkish-Gemma-9b-T1
    positive_prompts:
      - "matematiksel çözüm"
      - "python kodu yaz"
      - "algoritma kur"
      - "türev hesapla"
      - "javascript fonksiyonu"
      - "mantık sorusu"
      - "zincirleme düşünce"
      - "write python code"
      - "solve this equation"
      - "calculate derivative"
    negative_prompts:
      - "şiir yaz"
      - "roman anlat"
      - "günlük sohbet"
      - "write a poem"
      - "tell me a story"

  # EXPERT 1: THE SCHOLAR (Knowledge & Facts)
  - source_model: ytu-ce-cosmos/Turkish-Gemma-9b-v0.1
    positive_prompts:
      - "tarihsel olay nedir"
      - "biyolojik süreç"
      - "coğrafi bilgi"
      - "fizik kanunları"
      - "akademik makale özeti"
      - "what is the history of"
      - "scientific facts"
      - "laws of physics"
    negative_prompts:
      - "python kodu"
      - "hayal et"
      - "masal anlat"
      - "write code"

  # EXPERT 2: THE CREATIVE CHATBOT (Chat & Creative)
  - source_model: suayptalha/Sungur-9B
    positive_prompts:
      - "merhaba nasılsın"
      - "bana bir hikaye anlat"
      - "duygusal bir şiir yaz"
      - "rol yapma oyunu"
      - "arkadaşça sohbet"
      - "hello how are you"
      - "tell me a bedtime story"
      - "write a poem about love"
    negative_prompts:
      - "integral hesapla"
      - "c++ kodu"
      - "calculate integral"
      - "compile error"

</details><br>

Usage

To use this model, you need to set trust_remote_code=True.

AutoModelForCausalLM

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "suayptalha/Turkish-Gemma-2-MoE-3x9B"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

# Example: Reasoning (Routes to Expert 0 - Reasoner)
messages = [
    {"role": "user", "content": "5x + 1 = 16. x'i bul ve adımları açıkla."},
]

inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    do_sample=True,
    temperature=0.7
)

print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

Expert Selection Examples

Prompt (Logic): "Bir Python listesini sıralayan bubble sort algoritmasını yaz."

Activated Expert: The Reasoner (Expert 0)

Prompt (Knowledge): "Sir Isaac Newton kimdir?"

Activated Expert: The Scholar (Expert 1)

Prompt (Creative): "Yüzüklerin Efendisi evreninde geçen kısa, duygusal bir şiir yaz."

Activated Expert: The Creative (Expert 2 - Sungur-9B)

Acknowledgments

This model is a collaborative effort utilizing the best open-source Turkish models:

• Thanks to ytu-ce-cosmos for their amazing Turkish-Gemma-9b-v0.1 and Turkish-Gemma-9b-T1 models.
• Thanks to Google for the original Gemma 2 architecture and models.
• Gemma-2-MoE: The custom library I built to make this architecture possible.
• Thanks to all Turkish open source AI community.

Citation

@misc{sungur_collection_2025,
  title        = {Sungur (Hugging Face Collection)},
  author       = {Şuayp Talha Kocabay},
  year         = {2025},
  howpublished = {\url{[https://huggingface.co/collections/suayptalha/sungur-68dcd094da7f8976cdc5898e](https://huggingface.co/collections/suayptalha/sungur-68dcd094da7f8976cdc5898e)}},
  note         = {Turkish LLM family and dataset collection}
}

Support

<a href="https://www.buymeacoffee.com/suayptalha" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" style="height: 60px !important;width: 217px !important;" ></a>

---

license: gemma2