Turkish-Gemma-9b-T1-GGUF

<img src="./Turkish_gemma.png"/>

Turkish-Gemma-9b-T1

Turkish-Gemma-9b-T1 is based on ytu-ce-cosmos/Turkish-Gemma-9b-v0.1, adapted specifically for multi-step reasoning (“thinking”) in Turkish.

The model is designed to perform better at mathematical problems, logical reasoning, step-by-step inference, and planning tasks, while still following instructions to produce clear and concise final answers.

🚀 What’s New in the Reasoning Version?

• Multi-step reasoning: Stronger intermediate inference when multiple clues/conditions are involved.

• Math & logic: Improved accuracy on arithmetic, probability, sequences, rational reasoning, and logic puzzles.

• Better instruction following: Better adherence to prompts.

• Reduced hallucinations: The reasoning model hallucinates less, focusing on grounded answers and indicating uncertainty when necessary.

To evaluate model performance, we compiled a dataset of 1,450 carefully designed questions across diverse categories. Each question was reviewed and rated by 18 human annotators, allowing for a reliable comparison across multiple models.

The table below summarizes the evaluation results:

🏆 Model Comparison: Win Rates

Voting Metodology

A question and two answers from different models were presented to human judges. The judges selected the better answer based on their preferences. For example, in the question below, the judge evaluated both answers as good: <img src="./voting_new.png"/>

📊 Turkish Evaluation Gsm8k Benchmark Results

Note: When running Turkish evaluations on well-known benchmarks, it is important to adjust the evaluation configurations specifically for reasoning models. Default settings may not reflect the true performance, as factors like context handling and prompt formatting can significantly affect results. Carefully tuning these configs ensures fairer and more accurate comparisons across models.

Quick Start

The examples below demonstrate how to use the model to generate content based on given inputs.

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
import transformers

model_id = "ytu-ce-cosmos/Turkish-Gemma-9b-T1"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

pipeline = transformers.pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="auto",
)
messages = [
    {"role": "user", "content": "İstanbul halkı, timsahları evcilleştirip balkonlarda beslemeyi alışkanlık hale getirmiştir. Hangi timsah türleri en çok tercih edilir?"}
]
input_ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)
terminators = [
    pipeline.tokenizer.eos_token_id,
    pipeline.tokenizer.convert_tokens_to_ids("<end_of_turn>")
]
outputs = model.generate(
    input_ids,
    max_new_tokens=4096,
    eos_token_id=terminators,
    do_sample=False,
)
response = outputs[0][input_ids.shape[-1]:]
print(tokenizer.decode(response, skip_special_tokens=True))

## <think> .... </think>
## Bu soru gerçek dışı bir senaryo içeriyor. **İstanbul'da veya herhangi bir kentsel alanda timsahların evcilleştirilip balkonlarda beslenmesi mümkün değildir ve bu bir alışkanlık değildir.

Tips

Use Temperature=0.6, TopP=0.95, TopK=20, and MinP=0 (the default setting in generation_config.json). DO NOT use greedy decoding, as it can lead to performance degradation and endless repetitions.

• Complex tasks: Increase max_new_tokens. You can increase the repetition penalty and also adjust the presence_penalty parameter (between 0 and 2) to reduce endless repetitions. However, using a higher value may occasionally result in language mixing and a slight decrease in model performance

Acknowledgments

Thanks to Hugging Face for hosting models on S3 storage.

Compute resources were provided by the Barcelona Supercomputing Center

Contact

COSMOS AI Research Group – Yildiz Technical University, Computer Engineering Department
🔗 https://cosmos.yildiz.edu.tr/
✉️ [email protected]

---

license: gemma2